{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dd1475d8",
   "metadata": {
    "papermill": {
     "duration": 0.025999,
     "end_time": "2024-05-15T17:39:02.071896",
     "exception": false,
     "start_time": "2024-05-15T17:39:02.045897",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Import thư viện cần thiết"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4c756fc7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:02.120632Z",
     "iopub.status.busy": "2024-05-15T17:39:02.120215Z",
     "iopub.status.idle": "2024-05-15T17:39:04.777148Z",
     "shell.execute_reply": "2024-05-15T17:39:04.776145Z"
    },
    "papermill": {
     "duration": 2.683927,
     "end_time": "2024-05-15T17:39:04.780095",
     "exception": false,
     "start_time": "2024-05-15T17:39:02.096168",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a204c338",
   "metadata": {
    "papermill": {
     "duration": 0.022586,
     "end_time": "2024-05-15T17:39:04.826370",
     "exception": false,
     "start_time": "2024-05-15T17:39:04.803784",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Điều chính kích thước và độ phóng chung cho các biểu đồ trong bài tập này."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6ca45f79",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:04.876792Z",
     "iopub.status.busy": "2024-05-15T17:39:04.875573Z",
     "iopub.status.idle": "2024-05-15T17:39:04.880860Z",
     "shell.execute_reply": "2024-05-15T17:39:04.879773Z"
    },
    "papermill": {
     "duration": 0.033092,
     "end_time": "2024-05-15T17:39:04.883359",
     "exception": false,
     "start_time": "2024-05-15T17:39:04.850267",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "plt.rcParams['figure.figsize'] = [12, 7]\n",
    "plt.rcParams['figure.dpi'] = 300"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c16e813f",
   "metadata": {
    "papermill": {
     "duration": 0.022428,
     "end_time": "2024-05-15T17:39:04.929457",
     "exception": false,
     "start_time": "2024-05-15T17:39:04.907029",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Thu thập dữ liệu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e22db3f",
   "metadata": {
    "papermill": {
     "duration": 0.022512,
     "end_time": "2024-05-15T17:39:04.974460",
     "exception": false,
     "start_time": "2024-05-15T17:39:04.951948",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Chủ đề dữ liệu\n",
    "\n",
    "Dữ liệu trong cuộc thi \"Costa Rican Household Poverty Prediction\" trên Kaggle liên quan đến chủ đề xác định chính xác tiêu chuẩn thu nhập của các hộ gia đình nghèo tại Costa Rica. \n",
    "\n",
    "Câu chuyện cốt lõi: nhiều chương trình viện trợ xã hội gặp khó khăn và thách thức trong việc đảm bảo đúng người được viện trợ.\n",
    "\n",
    "Cuộc thi này nhằm mục đích phát triển các thuật toán và mô hình học máy dự đoán chính xác hơn giúp cải thiện hiệu suất để phân loại mức độ và nhu cầu của từng hộ gia đình. Từ đó, giúp các chương trình hỗ trợ xã hội cung cấp viện trợ hiệu quả hơn cho những hộ thật sự cần thiết tại Costa Rica."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33aa9994",
   "metadata": {
    "papermill": {
     "duration": 0.023805,
     "end_time": "2024-05-15T17:39:05.021257",
     "exception": false,
     "start_time": "2024-05-15T17:39:04.997452",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Mục đích lựa chọn dữ liệu\n",
    "\n",
    "Tăng cường nhận thức về vấn đề hỗ trợ những đối tượng khó khăn và đóng góp cách giải quyết bằng thuật toán tại Costa Rica nói riêng và các uqốc gia trên thế giới nói chung.\n",
    "\n",
    "Dữ liệu cũng giúp nhóm có cơ hội tiếp cận và làm việc với tập dữ liệu thực tế, qua đó cải thiện khả năng xử lý dữ liệu và kỹ năng xây dựng mô hình dự đoán."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f63a4830",
   "metadata": {
    "papermill": {
     "duration": 0.023152,
     "end_time": "2024-05-15T17:39:05.068055",
     "exception": false,
     "start_time": "2024-05-15T17:39:05.044903",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Cách thức thu thập dữ liệu\n",
    "\n",
    "Tập dữ liệu bao gồm các thông tin nhân khẩu, học vấn, công việc, thu nhập và các điều kiện sống khác. Dữ liệu được thu thập thông qua cuộc khảo sát trực tiếp tại các hộ gia đình hoặc thông qua các cơ quan tổ chức chính phủ hoặc phi chính phủ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24e9abc6",
   "metadata": {
    "papermill": {
     "duration": 0.023826,
     "end_time": "2024-05-15T17:39:05.115946",
     "exception": false,
     "start_time": "2024-05-15T17:39:05.092120",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Khám phá dữ liệu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af5aa6e2",
   "metadata": {
    "papermill": {
     "duration": 0.025896,
     "end_time": "2024-05-15T17:39:05.231910",
     "exception": false,
     "start_time": "2024-05-15T17:39:05.206014",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Đọc dữ liệu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1027a10a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:05.288479Z",
     "iopub.status.busy": "2024-05-15T17:39:05.288064Z",
     "iopub.status.idle": "2024-05-15T17:39:05.530458Z",
     "shell.execute_reply": "2024-05-15T17:39:05.529383Z"
    },
    "papermill": {
     "duration": 0.273364,
     "end_time": "2024-05-15T17:39:05.533080",
     "exception": false,
     "start_time": "2024-05-15T17:39:05.259716",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>v2a1</th>\n",
       "      <th>hacdor</th>\n",
       "      <th>rooms</th>\n",
       "      <th>hacapo</th>\n",
       "      <th>v14a</th>\n",
       "      <th>refrig</th>\n",
       "      <th>v18q</th>\n",
       "      <th>v18q1</th>\n",
       "      <th>r4h1</th>\n",
       "      <th>...</th>\n",
       "      <th>SQBescolari</th>\n",
       "      <th>SQBage</th>\n",
       "      <th>SQBhogar_total</th>\n",
       "      <th>SQBedjefe</th>\n",
       "      <th>SQBhogar_nin</th>\n",
       "      <th>SQBovercrowding</th>\n",
       "      <th>SQBdependency</th>\n",
       "      <th>SQBmeaned</th>\n",
       "      <th>agesq</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ID_279628684</td>\n",
       "      <td>190000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>100</td>\n",
       "      <td>1849</td>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>1849</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ID_f29eb3ddd</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>144</td>\n",
       "      <td>4489</td>\n",
       "      <td>1</td>\n",
       "      <td>144</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>64.0</td>\n",
       "      <td>144.0</td>\n",
       "      <td>4489</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ID_68de51c94</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>121</td>\n",
       "      <td>8464</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>64.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>8464</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ID_d671db89c</td>\n",
       "      <td>180000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>81</td>\n",
       "      <td>289</td>\n",
       "      <td>16</td>\n",
       "      <td>121</td>\n",
       "      <td>4</td>\n",
       "      <td>1.777778</td>\n",
       "      <td>1.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>289</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ID_d56d6f5f5</td>\n",
       "      <td>180000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>121</td>\n",
       "      <td>1369</td>\n",
       "      <td>16</td>\n",
       "      <td>121</td>\n",
       "      <td>4</td>\n",
       "      <td>1.777778</td>\n",
       "      <td>1.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>1369</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 143 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Id      v2a1  hacdor  rooms  hacapo  v14a  refrig  v18q  v18q1  \\\n",
       "0  ID_279628684  190000.0       0      3       0     1       1     0    NaN   \n",
       "1  ID_f29eb3ddd  135000.0       0      4       0     1       1     1    1.0   \n",
       "2  ID_68de51c94       NaN       0      8       0     1       1     0    NaN   \n",
       "3  ID_d671db89c  180000.0       0      5       0     1       1     1    1.0   \n",
       "4  ID_d56d6f5f5  180000.0       0      5       0     1       1     1    1.0   \n",
       "\n",
       "   r4h1  ...  SQBescolari  SQBage  SQBhogar_total  SQBedjefe  SQBhogar_nin  \\\n",
       "0     0  ...          100    1849               1        100             0   \n",
       "1     0  ...          144    4489               1        144             0   \n",
       "2     0  ...          121    8464               1          0             0   \n",
       "3     0  ...           81     289              16        121             4   \n",
       "4     0  ...          121    1369              16        121             4   \n",
       "\n",
       "   SQBovercrowding  SQBdependency  SQBmeaned  agesq  Target  \n",
       "0         1.000000            0.0      100.0   1849       4  \n",
       "1         1.000000           64.0      144.0   4489       4  \n",
       "2         0.250000           64.0      121.0   8464       4  \n",
       "3         1.777778            1.0      121.0    289       4  \n",
       "4         1.777778            1.0      121.0   1369       4  \n",
       "\n",
       "[5 rows x 143 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.read_csv('/kaggle/input/costa-rican-household-poverty-prediction/train.csv')\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ff7ba2e5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:05.585854Z",
     "iopub.status.busy": "2024-05-15T17:39:05.585446Z",
     "iopub.status.idle": "2024-05-15T17:39:06.008351Z",
     "shell.execute_reply": "2024-05-15T17:39:06.006854Z"
    },
    "papermill": {
     "duration": 0.452381,
     "end_time": "2024-05-15T17:39:06.011243",
     "exception": false,
     "start_time": "2024-05-15T17:39:05.558862",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>v2a1</th>\n",
       "      <th>hacdor</th>\n",
       "      <th>rooms</th>\n",
       "      <th>hacapo</th>\n",
       "      <th>v14a</th>\n",
       "      <th>refrig</th>\n",
       "      <th>v18q</th>\n",
       "      <th>v18q1</th>\n",
       "      <th>r4h1</th>\n",
       "      <th>...</th>\n",
       "      <th>age</th>\n",
       "      <th>SQBescolari</th>\n",
       "      <th>SQBage</th>\n",
       "      <th>SQBhogar_total</th>\n",
       "      <th>SQBedjefe</th>\n",
       "      <th>SQBhogar_nin</th>\n",
       "      <th>SQBovercrowding</th>\n",
       "      <th>SQBdependency</th>\n",
       "      <th>SQBmeaned</th>\n",
       "      <th>agesq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ID_2f6873615</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.25</td>\n",
       "      <td>0.25</td>\n",
       "      <td>272.25</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ID_1c78846d2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>41</td>\n",
       "      <td>256</td>\n",
       "      <td>1681</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.25</td>\n",
       "      <td>0.25</td>\n",
       "      <td>272.25</td>\n",
       "      <td>1681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ID_e5442cf6a</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>41</td>\n",
       "      <td>289</td>\n",
       "      <td>1681</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.25</td>\n",
       "      <td>0.25</td>\n",
       "      <td>272.25</td>\n",
       "      <td>1681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ID_a8db26a79</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>59</td>\n",
       "      <td>256</td>\n",
       "      <td>3481</td>\n",
       "      <td>1</td>\n",
       "      <td>256</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>256.00</td>\n",
       "      <td>3481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ID_a62966799</td>\n",
       "      <td>175000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>18</td>\n",
       "      <td>121</td>\n",
       "      <td>324</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.25</td>\n",
       "      <td>64.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>324</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 142 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Id      v2a1  hacdor  rooms  hacapo  v14a  refrig  v18q  v18q1  \\\n",
       "0  ID_2f6873615       NaN       0      5       0     1       1     0    NaN   \n",
       "1  ID_1c78846d2       NaN       0      5       0     1       1     0    NaN   \n",
       "2  ID_e5442cf6a       NaN       0      5       0     1       1     0    NaN   \n",
       "3  ID_a8db26a79       NaN       0     14       0     1       1     1    1.0   \n",
       "4  ID_a62966799  175000.0       0      4       0     1       1     1    1.0   \n",
       "\n",
       "   r4h1  ...  age  SQBescolari  SQBage  SQBhogar_total  SQBedjefe  \\\n",
       "0     1  ...    4            0      16               9          0   \n",
       "1     1  ...   41          256    1681               9          0   \n",
       "2     1  ...   41          289    1681               9          0   \n",
       "3     0  ...   59          256    3481               1        256   \n",
       "4     0  ...   18          121     324               1          0   \n",
       "\n",
       "   SQBhogar_nin  SQBovercrowding  SQBdependency  SQBmeaned  agesq  \n",
       "0             1             2.25           0.25     272.25     16  \n",
       "1             1             2.25           0.25     272.25   1681  \n",
       "2             1             2.25           0.25     272.25   1681  \n",
       "3             0             1.00           0.00     256.00   3481  \n",
       "4             1             0.25          64.00        NaN    324  \n",
       "\n",
       "[5 rows x 142 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = pd.read_csv('/kaggle/input/costa-rican-household-poverty-prediction/test.csv')\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed44e0cf",
   "metadata": {
    "papermill": {
     "duration": 0.027694,
     "end_time": "2024-05-15T17:39:06.066248",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.038554",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Số dòng và cột của dữ liệu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cce8b5f2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:06.119176Z",
     "iopub.status.busy": "2024-05-15T17:39:06.118805Z",
     "iopub.status.idle": "2024-05-15T17:39:06.125556Z",
     "shell.execute_reply": "2024-05-15T17:39:06.124304Z"
    },
    "papermill": {
     "duration": 0.037989,
     "end_time": "2024-05-15T17:39:06.128823",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.090834",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9557, 143)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_rows, n_col = train.shape\n",
    "n_rows, n_col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1103bf55",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:06.182164Z",
     "iopub.status.busy": "2024-05-15T17:39:06.181302Z",
     "iopub.status.idle": "2024-05-15T17:39:06.187811Z",
     "shell.execute_reply": "2024-05-15T17:39:06.186604Z"
    },
    "papermill": {
     "duration": 0.035739,
     "end_time": "2024-05-15T17:39:06.190238",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.154499",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(23856, 142)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_rows, n_col = test.shape\n",
    "n_rows, n_col"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c88b2c",
   "metadata": {
    "papermill": {
     "duration": 0.027337,
     "end_time": "2024-05-15T17:39:06.246482",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.219145",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Các dòng dữ liệu "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbca3765",
   "metadata": {
    "papermill": {
     "duration": 0.026017,
     "end_time": "2024-05-15T17:39:06.299905",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.273888",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Ý nghĩa mỗi dòng dữ liệu?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "168738fa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:06.356638Z",
     "iopub.status.busy": "2024-05-15T17:39:06.356258Z",
     "iopub.status.idle": "2024-05-15T17:39:06.390528Z",
     "shell.execute_reply": "2024-05-15T17:39:06.388658Z"
    },
    "papermill": {
     "duration": 0.066466,
     "end_time": "2024-05-15T17:39:06.393554",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.327088",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>v2a1</th>\n",
       "      <th>hacdor</th>\n",
       "      <th>rooms</th>\n",
       "      <th>hacapo</th>\n",
       "      <th>v14a</th>\n",
       "      <th>refrig</th>\n",
       "      <th>v18q</th>\n",
       "      <th>v18q1</th>\n",
       "      <th>r4h1</th>\n",
       "      <th>...</th>\n",
       "      <th>SQBescolari</th>\n",
       "      <th>SQBage</th>\n",
       "      <th>SQBhogar_total</th>\n",
       "      <th>SQBedjefe</th>\n",
       "      <th>SQBhogar_nin</th>\n",
       "      <th>SQBovercrowding</th>\n",
       "      <th>SQBdependency</th>\n",
       "      <th>SQBmeaned</th>\n",
       "      <th>agesq</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7818</th>\n",
       "      <td>ID_ae16aa0ac</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>1521</td>\n",
       "      <td>36</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>4.00</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>1521</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4973</th>\n",
       "      <td>ID_2013e8210</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1.777778</td>\n",
       "      <td>1.00</td>\n",
       "      <td>132.250000</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9324</th>\n",
       "      <td>ID_3fdd82459</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>49</td>\n",
       "      <td>289</td>\n",
       "      <td>9</td>\n",
       "      <td>36</td>\n",
       "      <td>1</td>\n",
       "      <td>2.250000</td>\n",
       "      <td>0.25</td>\n",
       "      <td>36.000000</td>\n",
       "      <td>289</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7467</th>\n",
       "      <td>ID_7d807f0e0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>36</td>\n",
       "      <td>1600</td>\n",
       "      <td>25</td>\n",
       "      <td>36</td>\n",
       "      <td>9</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>2.25</td>\n",
       "      <td>132.250000</td>\n",
       "      <td>1600</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8366</th>\n",
       "      <td>ID_79e84408d</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>4096</td>\n",
       "      <td>16</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.00</td>\n",
       "      <td>5.444444</td>\n",
       "      <td>4096</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3570</th>\n",
       "      <td>ID_f57bd173a</td>\n",
       "      <td>110000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>144</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>9.00</td>\n",
       "      <td>121.000000</td>\n",
       "      <td>144</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6422</th>\n",
       "      <td>ID_280e021d8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>36</td>\n",
       "      <td>2601</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>2601</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4823</th>\n",
       "      <td>ID_c96c9bf9e</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>144</td>\n",
       "      <td>2601</td>\n",
       "      <td>4</td>\n",
       "      <td>144</td>\n",
       "      <td>0</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>2601</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4782</th>\n",
       "      <td>ID_b4a68c897</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>7225</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.250000</td>\n",
       "      <td>4.00</td>\n",
       "      <td>0.444444</td>\n",
       "      <td>7225</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8487</th>\n",
       "      <td>ID_50db05a3a</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>121</td>\n",
       "      <td>784</td>\n",
       "      <td>16</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>52.562500</td>\n",
       "      <td>784</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 143 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Id      v2a1  hacdor  rooms  hacapo  v14a  refrig  v18q  \\\n",
       "7818  ID_ae16aa0ac       NaN       0      4       0     1       1     0   \n",
       "4973  ID_2013e8210       NaN       0      6       0     1       1     1   \n",
       "9324  ID_3fdd82459       NaN       0      4       0     1       1     0   \n",
       "7467  ID_7d807f0e0       NaN       0      5       0     1       1     0   \n",
       "8366  ID_79e84408d       NaN       0      4       0     1       1     0   \n",
       "3570  ID_f57bd173a  110000.0       0      4       0     1       1     0   \n",
       "6422  ID_280e021d8       NaN       0      4       0     1       1     0   \n",
       "4823  ID_c96c9bf9e       NaN       0      3       0     1       1     0   \n",
       "4782  ID_b4a68c897       NaN       0      2       0     1       1     0   \n",
       "8487  ID_50db05a3a       NaN       0      6       0     1       1     0   \n",
       "\n",
       "      v18q1  r4h1  ...  SQBescolari  SQBage  SQBhogar_total  SQBedjefe  \\\n",
       "7818    NaN     2  ...            9    1521              36          9   \n",
       "4973    2.0     1  ...            0       0              16          0   \n",
       "9324    NaN     0  ...           49     289               9         36   \n",
       "7467    NaN     1  ...           36    1600              25         36   \n",
       "8366    NaN     1  ...            9    4096              16          9   \n",
       "3570    NaN     0  ...            9     144              16          0   \n",
       "6422    NaN     0  ...           36    2601               4          0   \n",
       "4823    NaN     0  ...          144    2601               4        144   \n",
       "4782    NaN     0  ...            0    7225               9          0   \n",
       "8487    NaN     0  ...          121     784              16          9   \n",
       "\n",
       "      SQBhogar_nin  SQBovercrowding  SQBdependency   SQBmeaned  agesq  Target  \n",
       "7818            16         9.000000           4.00    9.000000   1521       2  \n",
       "4973             4         1.777778           1.00  132.250000      0       4  \n",
       "9324             1         2.250000           0.25   36.000000    289       4  \n",
       "7467             9         2.777778           2.25  132.250000   1600       2  \n",
       "8366             1         4.000000           1.00    5.444444   4096       1  \n",
       "3570             9         4.000000           9.00  121.000000    144       2  \n",
       "6422             0         4.000000           0.00    9.000000   2601       4  \n",
       "4823             0         4.000000           0.00   81.000000   2601       4  \n",
       "4782             0         2.250000           4.00    0.444444   7225       4  \n",
       "8487             0         1.000000           0.00   52.562500    784       4  \n",
       "\n",
       "[10 rows x 143 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64cbbc99",
   "metadata": {
    "papermill": {
     "duration": 0.028142,
     "end_time": "2024-05-15T17:39:06.446866",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.418724",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "+ Không nhận thấy những dòng có ý nghĩa khác nhau."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22496aee",
   "metadata": {
    "papermill": {
     "duration": 0.025308,
     "end_time": "2024-05-15T17:39:06.497814",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.472506",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Các dòng dữ liệu có trùng lặp không?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f7106884",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:06.550926Z",
     "iopub.status.busy": "2024-05-15T17:39:06.550518Z",
     "iopub.status.idle": "2024-05-15T17:39:06.593528Z",
     "shell.execute_reply": "2024-05-15T17:39:06.592046Z"
    },
    "papermill": {
     "duration": 0.072256,
     "end_time": "2024-05-15T17:39:06.596013",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.523757",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data have no duplicated row.\n"
     ]
    }
   ],
   "source": [
    "is_duplicate = train.duplicated().sum()\n",
    "if is_duplicate:\n",
    "    print(f\"Data have duplicated {is_duplicate} rows.\")\n",
    "else:\n",
    "    print(f\"Data have no duplicated row.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "253904b8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:06.647642Z",
     "iopub.status.busy": "2024-05-15T17:39:06.647187Z",
     "iopub.status.idle": "2024-05-15T17:39:06.716106Z",
     "shell.execute_reply": "2024-05-15T17:39:06.714735Z"
    },
    "papermill": {
     "duration": 0.097765,
     "end_time": "2024-05-15T17:39:06.718833",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.621068",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data have no duplicated row.\n"
     ]
    }
   ],
   "source": [
    "is_duplicate = test.duplicated().sum()\n",
    "if is_duplicate:\n",
    "    print(f\"Data have duplicated {is_duplicate} rows.\")\n",
    "else:\n",
    "    print(f\"Data have no duplicated row.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "805f4154",
   "metadata": {
    "papermill": {
     "duration": 0.025847,
     "end_time": "2024-05-15T17:39:06.769863",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.744016",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Các cột dữ liệu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bfddea1",
   "metadata": {
    "papermill": {
     "duration": 0.025438,
     "end_time": "2024-05-15T17:39:06.822607",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.797169",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Ý nghĩa các cột"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9f565e70",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:06.874441Z",
     "iopub.status.busy": "2024-05-15T17:39:06.873671Z",
     "iopub.status.idle": "2024-05-15T17:39:06.897563Z",
     "shell.execute_reply": "2024-05-15T17:39:06.896330Z"
    },
    "papermill": {
     "duration": 0.052656,
     "end_time": "2024-05-15T17:39:06.900030",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.847374",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Variable name</th>\n",
       "      <th>Variable description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>v2a1</td>\n",
       "      <td>Monthly rent payment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>hacdor</td>\n",
       "      <td>=1 Overcrowding by bedrooms</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>rooms</td>\n",
       "      <td>number of all rooms in the house</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>hacapo</td>\n",
       "      <td>=1 Overcrowding by rooms</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>v14a</td>\n",
       "      <td>=1 has toilet in the household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>refrig</td>\n",
       "      <td>=1 if the household has refrigerator</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>v18q</td>\n",
       "      <td>owns a tablet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>v18q1</td>\n",
       "      <td>number of tablets household owns</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>r4h1</td>\n",
       "      <td>Males younger than 12 years of age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>r4h2</td>\n",
       "      <td>Males 12 years of age and older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>r4h3</td>\n",
       "      <td>Total males in the household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>r4m1</td>\n",
       "      <td>Females younger than 12 years of age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>r4m2</td>\n",
       "      <td>Females 12 years of age and older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>r4m3</td>\n",
       "      <td>Total females in the household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>r4t1</td>\n",
       "      <td>persons younger than 12 years of age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>r4t2</td>\n",
       "      <td>persons 12 years of age and older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>r4t3</td>\n",
       "      <td>Total persons in the household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>tamhog</td>\n",
       "      <td>size of the household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>tamviv</td>\n",
       "      <td>TamViv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>escolari</td>\n",
       "      <td>years of schooling</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>rez_esc</td>\n",
       "      <td>Years behind in school</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>hhsize</td>\n",
       "      <td>household size</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>paredblolad</td>\n",
       "      <td>=1 if predominant material on the outside wall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>paredzocalo</td>\n",
       "      <td>=1 if predominant material on the outside wall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>paredpreb</td>\n",
       "      <td>=1 if predominant material on the outside wall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>pareddes</td>\n",
       "      <td>=1 if predominant material on the outside wall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>paredmad</td>\n",
       "      <td>=1 if predominant material on the outside wall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>paredzinc</td>\n",
       "      <td>=1 if predominant material on the outside wall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>paredfibras</td>\n",
       "      <td>=1 if predominant material on the outside wall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>paredother</td>\n",
       "      <td>=1 if predominant material on the outside wall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>pisomoscer</td>\n",
       "      <td>=1 if predominant material on the floor is mos...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>pisocemento</td>\n",
       "      <td>=1 if predominant material on the floor is cement</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>pisoother</td>\n",
       "      <td>=1 if predominant material on the floor is other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>pisonatur</td>\n",
       "      <td>=1 if predominant material on the floor is  na...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>pisonotiene</td>\n",
       "      <td>=1 if no floor at the household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>pisomadera</td>\n",
       "      <td>=1 if predominant material on the floor is wood</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>techozinc</td>\n",
       "      <td>=1 if predominant material on the roof is meta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>techoentrepiso</td>\n",
       "      <td>=1 if predominant material on the roof is fibe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>techocane</td>\n",
       "      <td>=1 if predominant material on the roof is natu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>techootro</td>\n",
       "      <td>=1 if predominant material on the roof is other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>cielorazo</td>\n",
       "      <td>=1 if the house has ceiling</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>abastaguadentro</td>\n",
       "      <td>=1 if water provision inside the dwelling</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>abastaguafuera</td>\n",
       "      <td>=1 if water provision outside the dwelling</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>abastaguano</td>\n",
       "      <td>=1 if no water provision</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>public</td>\n",
       "      <td>=1 electricity from CNFL, ICE, ESPH/JASEC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>planpri</td>\n",
       "      <td>=1 electricity from private plant</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>noelec</td>\n",
       "      <td>=1 no electricity in the dwelling</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>coopele</td>\n",
       "      <td>=1 electricity from cooperative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>sanitario1</td>\n",
       "      <td>=1 no toilet in the dwelling</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>sanitario2</td>\n",
       "      <td>=1 toilet connected to sewer or cesspool</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>sanitario3</td>\n",
       "      <td>=1 toilet connected to  septic tank</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>sanitario5</td>\n",
       "      <td>=1 toilet connected to black hole or letrine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>sanitario6</td>\n",
       "      <td>=1 toilet connected to other system</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>energcocinar1</td>\n",
       "      <td>=1 no main source of energy used for cooking (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>energcocinar2</td>\n",
       "      <td>=1 main source of energy used for cooking elec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>energcocinar3</td>\n",
       "      <td>=1 main source of energy used for cooking gas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>energcocinar4</td>\n",
       "      <td>=1 main source of energy used for cooking wood...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>elimbasu1</td>\n",
       "      <td>=1 if rubbish disposal mainly by tanker truck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>elimbasu2</td>\n",
       "      <td>=1 if rubbish disposal mainly by botan hollow ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>elimbasu3</td>\n",
       "      <td>=1 if rubbish disposal mainly by burning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>elimbasu4</td>\n",
       "      <td>=1 if rubbish disposal mainly by throwing in a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>elimbasu5</td>\n",
       "      <td>=1 if rubbish disposal mainly by throwing in r...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>elimbasu6</td>\n",
       "      <td>=1 if rubbish disposal mainly other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>epared1</td>\n",
       "      <td>=1 if walls are bad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>epared2</td>\n",
       "      <td>=1 if walls are regular</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>epared3</td>\n",
       "      <td>=1 if walls are good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>etecho1</td>\n",
       "      <td>=1 if roof are bad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>etecho2</td>\n",
       "      <td>=1 if roof are regular</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>etecho3</td>\n",
       "      <td>=1 if roof are good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>eviv1</td>\n",
       "      <td>=1 if floor are bad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>eviv2</td>\n",
       "      <td>=1 if floor are regular</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>eviv3</td>\n",
       "      <td>=1 if floor are good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>dis</td>\n",
       "      <td>=1 if disable person</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>male</td>\n",
       "      <td>=1 if male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>female</td>\n",
       "      <td>=1 if female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>estadocivil1</td>\n",
       "      <td>=1 if less than 10 years old</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>estadocivil2</td>\n",
       "      <td>=1 if free or coupled uunion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>estadocivil3</td>\n",
       "      <td>=1 if married</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>estadocivil4</td>\n",
       "      <td>=1 if divorced</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>estadocivil5</td>\n",
       "      <td>=1 if separated</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>estadocivil6</td>\n",
       "      <td>=1 if widow/er</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>estadocivil7</td>\n",
       "      <td>=1 if single</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>parentesco1</td>\n",
       "      <td>=1 if household head</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>parentesco2</td>\n",
       "      <td>=1 if spouse/partner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>parentesco3</td>\n",
       "      <td>=1 if son/doughter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>parentesco4</td>\n",
       "      <td>=1 if stepson/doughter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>parentesco5</td>\n",
       "      <td>=1 if son/doughter in law</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>parentesco6</td>\n",
       "      <td>=1 if grandson/doughter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>parentesco7</td>\n",
       "      <td>=1 if mother/father</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>parentesco8</td>\n",
       "      <td>=1 if father/mother in law</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>parentesco9</td>\n",
       "      <td>=1 if brother/sister</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>parentesco10</td>\n",
       "      <td>=1 if brother/sister in law</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>parentesco11</td>\n",
       "      <td>=1 if other family member</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>parentesco12</td>\n",
       "      <td>=1 if other non family member</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>idhogar</td>\n",
       "      <td>Household level identifier</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>hogar_nin</td>\n",
       "      <td>Number of children 0 to 19 in household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>hogar_adul</td>\n",
       "      <td>Number of adults in household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>hogar_mayor</td>\n",
       "      <td># of individuals 65+ in the household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>hogar_total</td>\n",
       "      <td># of total individuals in the household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>dependency</td>\n",
       "      <td>Dependency rate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>edjefe</td>\n",
       "      <td>years of education of male head of household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>edjefa</td>\n",
       "      <td>years of education of female head of household</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>meaneduc</td>\n",
       "      <td>average years of education for adults (18+)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>instlevel1</td>\n",
       "      <td>=1 no level of education</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>instlevel2</td>\n",
       "      <td>=1 incomplete primary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>instlevel3</td>\n",
       "      <td>=1 complete primary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>instlevel4</td>\n",
       "      <td>=1 incomplete academic secondary level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>instlevel5</td>\n",
       "      <td>=1 complete academic secondary level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>instlevel6</td>\n",
       "      <td>=1 incomplete technical secondary level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>instlevel7</td>\n",
       "      <td>=1 complete technical secondary level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>instlevel8</td>\n",
       "      <td>=1 undergraduate and higher education</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>instlevel9</td>\n",
       "      <td>=1 postgraduate higher education</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>bedrooms</td>\n",
       "      <td>number of bedrooms</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>overcrowding</td>\n",
       "      <td># persons per room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>tipovivi1</td>\n",
       "      <td>=1 own and fully paid house</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>tipovivi2</td>\n",
       "      <td>=1 own, paying in installments</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>tipovivi3</td>\n",
       "      <td>=1 rented</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>tipovivi4</td>\n",
       "      <td>=1 precarious</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>tipovivi5</td>\n",
       "      <td>=1 other(assigned, borrowed)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>computer</td>\n",
       "      <td>=1 if the household has notebook or desktop co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>television</td>\n",
       "      <td>=1 if the household has TV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>mobilephone</td>\n",
       "      <td>=1 if mobile phone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>qmobilephone</td>\n",
       "      <td># of mobile phones</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>lugar1</td>\n",
       "      <td>=1 region Central</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>lugar2</td>\n",
       "      <td>=1 region Chorotega</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>lugar3</td>\n",
       "      <td>=1 region PacÃƒÂ­fico central</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>lugar4</td>\n",
       "      <td>=1 region Brunca</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>lugar5</td>\n",
       "      <td>=1 region Huetar AtlÃƒÂ¡ntica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>lugar6</td>\n",
       "      <td>=1 region Huetar Norte</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>area1</td>\n",
       "      <td>=1 zona urbana</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>area2</td>\n",
       "      <td>=2 zona rural</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>age</td>\n",
       "      <td>Age in years</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>SQBescolari</td>\n",
       "      <td>escolari squared</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133</th>\n",
       "      <td>SQBage</td>\n",
       "      <td>age squared</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>SQBhogar_total</td>\n",
       "      <td>hogar_total squared</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>SQBedjefe</td>\n",
       "      <td>edjefe squared</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>SQBhogar_nin</td>\n",
       "      <td>hogar_nin squared</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>SQBovercrowding</td>\n",
       "      <td>overcrowding squared</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>SQBdependency</td>\n",
       "      <td>dependency squared</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>SQBmeaned</td>\n",
       "      <td>meaned squared</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>agesq</td>\n",
       "      <td>Age squared</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Variable name                               Variable description\n",
       "0               v2a1                               Monthly rent payment\n",
       "1             hacdor                        =1 Overcrowding by bedrooms\n",
       "2              rooms                   number of all rooms in the house\n",
       "3             hacapo                           =1 Overcrowding by rooms\n",
       "4               v14a                     =1 has toilet in the household\n",
       "5             refrig               =1 if the household has refrigerator\n",
       "6               v18q                                      owns a tablet\n",
       "7              v18q1                   number of tablets household owns\n",
       "8               r4h1                 Males younger than 12 years of age\n",
       "9               r4h2                    Males 12 years of age and older\n",
       "10              r4h3                       Total males in the household\n",
       "11              r4m1               Females younger than 12 years of age\n",
       "12              r4m2                  Females 12 years of age and older\n",
       "13              r4m3                     Total females in the household\n",
       "14              r4t1               persons younger than 12 years of age\n",
       "15              r4t2                  persons 12 years of age and older\n",
       "16              r4t3                     Total persons in the household\n",
       "17            tamhog                              size of the household\n",
       "18            tamviv                                             TamViv\n",
       "19          escolari                                 years of schooling\n",
       "20           rez_esc                             Years behind in school\n",
       "21            hhsize                                     household size\n",
       "22       paredblolad  =1 if predominant material on the outside wall...\n",
       "23       paredzocalo  =1 if predominant material on the outside wall...\n",
       "24         paredpreb  =1 if predominant material on the outside wall...\n",
       "25          pareddes  =1 if predominant material on the outside wall...\n",
       "26          paredmad  =1 if predominant material on the outside wall...\n",
       "27         paredzinc  =1 if predominant material on the outside wall...\n",
       "28       paredfibras  =1 if predominant material on the outside wall...\n",
       "29        paredother  =1 if predominant material on the outside wall...\n",
       "30        pisomoscer  =1 if predominant material on the floor is mos...\n",
       "31       pisocemento  =1 if predominant material on the floor is cement\n",
       "32         pisoother   =1 if predominant material on the floor is other\n",
       "33         pisonatur  =1 if predominant material on the floor is  na...\n",
       "34       pisonotiene                    =1 if no floor at the household\n",
       "35        pisomadera    =1 if predominant material on the floor is wood\n",
       "36         techozinc  =1 if predominant material on the roof is meta...\n",
       "37    techoentrepiso  =1 if predominant material on the roof is fibe...\n",
       "38         techocane  =1 if predominant material on the roof is natu...\n",
       "39         techootro    =1 if predominant material on the roof is other\n",
       "40         cielorazo                        =1 if the house has ceiling\n",
       "41   abastaguadentro          =1 if water provision inside the dwelling\n",
       "42    abastaguafuera         =1 if water provision outside the dwelling\n",
       "43       abastaguano                           =1 if no water provision\n",
       "44            public          =1 electricity from CNFL, ICE, ESPH/JASEC\n",
       "45           planpri                  =1 electricity from private plant\n",
       "46            noelec                  =1 no electricity in the dwelling\n",
       "47           coopele                    =1 electricity from cooperative\n",
       "48        sanitario1                       =1 no toilet in the dwelling\n",
       "49        sanitario2           =1 toilet connected to sewer or cesspool\n",
       "50        sanitario3                =1 toilet connected to  septic tank\n",
       "51        sanitario5       =1 toilet connected to black hole or letrine\n",
       "52        sanitario6                =1 toilet connected to other system\n",
       "53     energcocinar1  =1 no main source of energy used for cooking (...\n",
       "54     energcocinar2  =1 main source of energy used for cooking elec...\n",
       "55     energcocinar3      =1 main source of energy used for cooking gas\n",
       "56     energcocinar4  =1 main source of energy used for cooking wood...\n",
       "57         elimbasu1      =1 if rubbish disposal mainly by tanker truck\n",
       "58         elimbasu2  =1 if rubbish disposal mainly by botan hollow ...\n",
       "59         elimbasu3           =1 if rubbish disposal mainly by burning\n",
       "60         elimbasu4  =1 if rubbish disposal mainly by throwing in a...\n",
       "61         elimbasu5  =1 if rubbish disposal mainly by throwing in r...\n",
       "62         elimbasu6                =1 if rubbish disposal mainly other\n",
       "63           epared1                                =1 if walls are bad\n",
       "64           epared2                            =1 if walls are regular\n",
       "65           epared3                               =1 if walls are good\n",
       "66           etecho1                                 =1 if roof are bad\n",
       "67           etecho2                             =1 if roof are regular\n",
       "68           etecho3                                =1 if roof are good\n",
       "69             eviv1                                =1 if floor are bad\n",
       "70             eviv2                            =1 if floor are regular\n",
       "71             eviv3                               =1 if floor are good\n",
       "72               dis                               =1 if disable person\n",
       "73              male                                         =1 if male\n",
       "74            female                                       =1 if female\n",
       "75      estadocivil1                       =1 if less than 10 years old\n",
       "76      estadocivil2                       =1 if free or coupled uunion\n",
       "77      estadocivil3                                      =1 if married\n",
       "78      estadocivil4                                     =1 if divorced\n",
       "79      estadocivil5                                    =1 if separated\n",
       "80      estadocivil6                                     =1 if widow/er\n",
       "81      estadocivil7                                       =1 if single\n",
       "82       parentesco1                               =1 if household head\n",
       "83       parentesco2                               =1 if spouse/partner\n",
       "84       parentesco3                                 =1 if son/doughter\n",
       "85       parentesco4                             =1 if stepson/doughter\n",
       "86       parentesco5                          =1 if son/doughter in law\n",
       "87       parentesco6                            =1 if grandson/doughter\n",
       "88       parentesco7                                =1 if mother/father\n",
       "89       parentesco8                         =1 if father/mother in law\n",
       "90       parentesco9                               =1 if brother/sister\n",
       "91      parentesco10                        =1 if brother/sister in law\n",
       "92      parentesco11                          =1 if other family member\n",
       "93      parentesco12                      =1 if other non family member\n",
       "94           idhogar                         Household level identifier\n",
       "95         hogar_nin            Number of children 0 to 19 in household\n",
       "96        hogar_adul                      Number of adults in household\n",
       "97       hogar_mayor              # of individuals 65+ in the household\n",
       "98       hogar_total            # of total individuals in the household\n",
       "99        dependency                                    Dependency rate\n",
       "100           edjefe       years of education of male head of household\n",
       "101           edjefa     years of education of female head of household\n",
       "102         meaneduc        average years of education for adults (18+)\n",
       "103       instlevel1                           =1 no level of education\n",
       "104       instlevel2                              =1 incomplete primary\n",
       "105       instlevel3                                =1 complete primary\n",
       "106       instlevel4             =1 incomplete academic secondary level\n",
       "107       instlevel5               =1 complete academic secondary level\n",
       "108       instlevel6            =1 incomplete technical secondary level\n",
       "109       instlevel7              =1 complete technical secondary level\n",
       "110       instlevel8              =1 undergraduate and higher education\n",
       "111       instlevel9                   =1 postgraduate higher education\n",
       "112         bedrooms                                 number of bedrooms\n",
       "113     overcrowding                                 # persons per room\n",
       "114        tipovivi1                        =1 own and fully paid house\n",
       "115        tipovivi2                     =1 own, paying in installments\n",
       "116        tipovivi3                                          =1 rented\n",
       "117        tipovivi4                                      =1 precarious\n",
       "118        tipovivi5                       =1 other(assigned, borrowed)\n",
       "119         computer  =1 if the household has notebook or desktop co...\n",
       "120       television                         =1 if the household has TV\n",
       "121      mobilephone                                 =1 if mobile phone\n",
       "122     qmobilephone                                 # of mobile phones\n",
       "123           lugar1                                  =1 region Central\n",
       "124           lugar2                                =1 region Chorotega\n",
       "125           lugar3                      =1 region PacÃƒÂ­fico central\n",
       "126           lugar4                                   =1 region Brunca\n",
       "127           lugar5                      =1 region Huetar AtlÃƒÂ¡ntica\n",
       "128           lugar6                             =1 region Huetar Norte\n",
       "129            area1                                     =1 zona urbana\n",
       "130            area2                                      =2 zona rural\n",
       "131              age                                       Age in years\n",
       "132      SQBescolari                                   escolari squared\n",
       "133           SQBage                                        age squared\n",
       "134   SQBhogar_total                                hogar_total squared\n",
       "135        SQBedjefe                                     edjefe squared\n",
       "136     SQBhogar_nin                                  hogar_nin squared\n",
       "137  SQBovercrowding                               overcrowding squared\n",
       "138    SQBdependency                                 dependency squared\n",
       "139        SQBmeaned                                     meaned squared\n",
       "140            agesq                                        Age squared"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_rows', None)\n",
    "meaning = pd.read_csv('/kaggle/input/costa-rican-household-poverty-prediction/codebook.csv')\n",
    "meaning "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a2361c4",
   "metadata": {
    "papermill": {
     "duration": 0.025934,
     "end_time": "2024-05-15T17:39:06.953056",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.927122",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Thông tin các cột có thể đọc thêm tại [trang web](https://www.kaggle.com/competitions/costa-rican-household-poverty-prediction/data) của tập dữ liệu trên Kaggle."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00869948",
   "metadata": {
    "papermill": {
     "duration": 0.026615,
     "end_time": "2024-05-15T17:39:07.008272",
     "exception": false,
     "start_time": "2024-05-15T17:39:06.981657",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Kiểu dữ liệu của các cột"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "632b6b41",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:07.062733Z",
     "iopub.status.busy": "2024-05-15T17:39:07.062318Z",
     "iopub.status.idle": "2024-05-15T17:39:07.073997Z",
     "shell.execute_reply": "2024-05-15T17:39:07.073198Z"
    },
    "papermill": {
     "duration": 0.042722,
     "end_time": "2024-05-15T17:39:07.077771",
     "exception": false,
     "start_time": "2024-05-15T17:39:07.035049",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                  object\n",
       "v2a1               float64\n",
       "hacdor               int64\n",
       "rooms                int64\n",
       "hacapo               int64\n",
       "v14a                 int64\n",
       "refrig               int64\n",
       "v18q                 int64\n",
       "v18q1              float64\n",
       "r4h1                 int64\n",
       "r4h2                 int64\n",
       "r4h3                 int64\n",
       "r4m1                 int64\n",
       "r4m2                 int64\n",
       "r4m3                 int64\n",
       "r4t1                 int64\n",
       "r4t2                 int64\n",
       "r4t3                 int64\n",
       "tamhog               int64\n",
       "tamviv               int64\n",
       "escolari             int64\n",
       "rez_esc            float64\n",
       "hhsize               int64\n",
       "paredblolad          int64\n",
       "paredzocalo          int64\n",
       "paredpreb            int64\n",
       "pareddes             int64\n",
       "paredmad             int64\n",
       "paredzinc            int64\n",
       "paredfibras          int64\n",
       "paredother           int64\n",
       "pisomoscer           int64\n",
       "pisocemento          int64\n",
       "pisoother            int64\n",
       "pisonatur            int64\n",
       "pisonotiene          int64\n",
       "pisomadera           int64\n",
       "techozinc            int64\n",
       "techoentrepiso       int64\n",
       "techocane            int64\n",
       "techootro            int64\n",
       "cielorazo            int64\n",
       "abastaguadentro      int64\n",
       "abastaguafuera       int64\n",
       "abastaguano          int64\n",
       "public               int64\n",
       "planpri              int64\n",
       "noelec               int64\n",
       "coopele              int64\n",
       "sanitario1           int64\n",
       "sanitario2           int64\n",
       "sanitario3           int64\n",
       "sanitario5           int64\n",
       "sanitario6           int64\n",
       "energcocinar1        int64\n",
       "energcocinar2        int64\n",
       "energcocinar3        int64\n",
       "energcocinar4        int64\n",
       "elimbasu1            int64\n",
       "elimbasu2            int64\n",
       "elimbasu3            int64\n",
       "elimbasu4            int64\n",
       "elimbasu5            int64\n",
       "elimbasu6            int64\n",
       "epared1              int64\n",
       "epared2              int64\n",
       "epared3              int64\n",
       "etecho1              int64\n",
       "etecho2              int64\n",
       "etecho3              int64\n",
       "eviv1                int64\n",
       "eviv2                int64\n",
       "eviv3                int64\n",
       "dis                  int64\n",
       "male                 int64\n",
       "female               int64\n",
       "estadocivil1         int64\n",
       "estadocivil2         int64\n",
       "estadocivil3         int64\n",
       "estadocivil4         int64\n",
       "estadocivil5         int64\n",
       "estadocivil6         int64\n",
       "estadocivil7         int64\n",
       "parentesco1          int64\n",
       "parentesco2          int64\n",
       "parentesco3          int64\n",
       "parentesco4          int64\n",
       "parentesco5          int64\n",
       "parentesco6          int64\n",
       "parentesco7          int64\n",
       "parentesco8          int64\n",
       "parentesco9          int64\n",
       "parentesco10         int64\n",
       "parentesco11         int64\n",
       "parentesco12         int64\n",
       "idhogar             object\n",
       "hogar_nin            int64\n",
       "hogar_adul           int64\n",
       "hogar_mayor          int64\n",
       "hogar_total          int64\n",
       "dependency          object\n",
       "edjefe              object\n",
       "edjefa              object\n",
       "meaneduc           float64\n",
       "instlevel1           int64\n",
       "instlevel2           int64\n",
       "instlevel3           int64\n",
       "instlevel4           int64\n",
       "instlevel5           int64\n",
       "instlevel6           int64\n",
       "instlevel7           int64\n",
       "instlevel8           int64\n",
       "instlevel9           int64\n",
       "bedrooms             int64\n",
       "overcrowding       float64\n",
       "tipovivi1            int64\n",
       "tipovivi2            int64\n",
       "tipovivi3            int64\n",
       "tipovivi4            int64\n",
       "tipovivi5            int64\n",
       "computer             int64\n",
       "television           int64\n",
       "mobilephone          int64\n",
       "qmobilephone         int64\n",
       "lugar1               int64\n",
       "lugar2               int64\n",
       "lugar3               int64\n",
       "lugar4               int64\n",
       "lugar5               int64\n",
       "lugar6               int64\n",
       "area1                int64\n",
       "area2                int64\n",
       "age                  int64\n",
       "SQBescolari          int64\n",
       "SQBage               int64\n",
       "SQBhogar_total       int64\n",
       "SQBedjefe            int64\n",
       "SQBhogar_nin         int64\n",
       "SQBovercrowding    float64\n",
       "SQBdependency      float64\n",
       "SQBmeaned          float64\n",
       "agesq                int64\n",
       "Target               int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38f12f2d",
   "metadata": {
    "papermill": {
     "duration": 0.027909,
     "end_time": "2024-05-15T17:39:07.133912",
     "exception": false,
     "start_time": "2024-05-15T17:39:07.106003",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Các cột của tập dữ liệu `test` cũng tương tự."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48748ae6",
   "metadata": {
    "papermill": {
     "duration": 0.027924,
     "end_time": "2024-05-15T17:39:07.192030",
     "exception": false,
     "start_time": "2024-05-15T17:39:07.164106",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Với các cột numeric, các giá trị được phân bố như thế nào?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fa7f4b2",
   "metadata": {
    "papermill": {
     "duration": 0.032465,
     "end_time": "2024-05-15T17:39:07.253651",
     "exception": false,
     "start_time": "2024-05-15T17:39:07.221186",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Ở phần này, với các cột có kiểu dữ liệu `numeric`, ta sẽ tính:\n",
    "+ Tỉ lệ phần trăm các giá trị thiếu.\n",
    "+ Giá trị min, mean, max.\n",
    "+ Giá trị tứ phân vị thứ nhất, trung vị và tứ phân vị thứ 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9a9e6e3b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:07.318053Z",
     "iopub.status.busy": "2024-05-15T17:39:07.317442Z",
     "iopub.status.idle": "2024-05-15T17:39:08.158008Z",
     "shell.execute_reply": "2024-05-15T17:39:08.156573Z"
    },
    "papermill": {
     "duration": 0.871173,
     "end_time": "2024-05-15T17:39:08.160610",
     "exception": false,
     "start_time": "2024-05-15T17:39:07.289437",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>v2a1</th>\n",
       "      <th>hacdor</th>\n",
       "      <th>rooms</th>\n",
       "      <th>hacapo</th>\n",
       "      <th>v14a</th>\n",
       "      <th>refrig</th>\n",
       "      <th>v18q</th>\n",
       "      <th>v18q1</th>\n",
       "      <th>r4h1</th>\n",
       "      <th>r4h2</th>\n",
       "      <th>...</th>\n",
       "      <th>SQBescolari</th>\n",
       "      <th>SQBage</th>\n",
       "      <th>SQBhogar_total</th>\n",
       "      <th>SQBedjefe</th>\n",
       "      <th>SQBhogar_nin</th>\n",
       "      <th>SQBovercrowding</th>\n",
       "      <th>SQBdependency</th>\n",
       "      <th>SQBmeaned</th>\n",
       "      <th>agesq</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>missing_ratio</th>\n",
       "      <td>71.8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>76.8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>165231.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.6</td>\n",
       "      <td>...</td>\n",
       "      <td>74.2</td>\n",
       "      <td>1643.8</td>\n",
       "      <td>19.1</td>\n",
       "      <td>53.5</td>\n",
       "      <td>3.8</td>\n",
       "      <td>3.20</td>\n",
       "      <td>3.9</td>\n",
       "      <td>102.6</td>\n",
       "      <td>1643.8</td>\n",
       "      <td>3.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lower_quartile</th>\n",
       "      <td>80000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>289.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.1</td>\n",
       "      <td>36.0</td>\n",
       "      <td>289.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>median</th>\n",
       "      <td>130000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>36.0</td>\n",
       "      <td>961.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.20</td>\n",
       "      <td>0.4</td>\n",
       "      <td>81.0</td>\n",
       "      <td>961.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>upper_quartile</th>\n",
       "      <td>200000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>121.0</td>\n",
       "      <td>2601.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.00</td>\n",
       "      <td>1.8</td>\n",
       "      <td>134.6</td>\n",
       "      <td>2601.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2353477.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>...</td>\n",
       "      <td>441.0</td>\n",
       "      <td>9409.0</td>\n",
       "      <td>169.0</td>\n",
       "      <td>441.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>36.00</td>\n",
       "      <td>64.0</td>\n",
       "      <td>1369.0</td>\n",
       "      <td>9409.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7 rows × 138 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     v2a1  hacdor  rooms  hacapo  v14a  refrig  v18q  v18q1  \\\n",
       "missing_ratio        71.8     0.0    0.0     0.0   0.0     0.0   0.0   76.8   \n",
       "min                   0.0     0.0    1.0     0.0   0.0     0.0   0.0    1.0   \n",
       "mean             165231.6     0.0    5.0     0.0   1.0     1.0   0.2    1.4   \n",
       "lower_quartile    80000.0     0.0    4.0     0.0   1.0     1.0   0.0    1.0   \n",
       "median           130000.0     0.0    5.0     0.0   1.0     1.0   0.0    1.0   \n",
       "upper_quartile   200000.0     0.0    6.0     0.0   1.0     1.0   0.0    2.0   \n",
       "max             2353477.0     1.0   11.0     1.0   1.0     1.0   1.0    6.0   \n",
       "\n",
       "                r4h1  r4h2  ...  SQBescolari  SQBage  SQBhogar_total  \\\n",
       "missing_ratio    0.0   0.0  ...          0.0     0.0             0.0   \n",
       "min              0.0   0.0  ...          0.0     0.0             1.0   \n",
       "mean             0.4   1.6  ...         74.2  1643.8            19.1   \n",
       "lower_quartile   0.0   1.0  ...         16.0   289.0             9.0   \n",
       "median           0.0   1.0  ...         36.0   961.0            16.0   \n",
       "upper_quartile   1.0   2.0  ...        121.0  2601.0            25.0   \n",
       "max              5.0   8.0  ...        441.0  9409.0           169.0   \n",
       "\n",
       "                SQBedjefe  SQBhogar_nin  SQBovercrowding  SQBdependency  \\\n",
       "missing_ratio         0.0           0.0             0.00            0.0   \n",
       "min                   0.0           0.0             0.04            0.0   \n",
       "mean                 53.5           3.8             3.20            3.9   \n",
       "lower_quartile        0.0           0.0             1.00            0.1   \n",
       "median               36.0           1.0             2.20            0.4   \n",
       "upper_quartile       81.0           4.0             4.00            1.8   \n",
       "max                 441.0          81.0            36.00           64.0   \n",
       "\n",
       "                SQBmeaned   agesq  Target  \n",
       "missing_ratio         0.1     0.0     0.0  \n",
       "min                   0.0     0.0     1.0  \n",
       "mean                102.6  1643.8     3.3  \n",
       "lower_quartile       36.0   289.0     3.0  \n",
       "median               81.0   961.0     4.0  \n",
       "upper_quartile      134.6  2601.0     4.0  \n",
       "max                1369.0  9409.0     4.0  \n",
       "\n",
       "[7 rows x 138 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def missing_ratio(df):\n",
    "    return (df.isnull().sum() * 100 / len(df)).round(1)\n",
    "\n",
    "def lower_quartile(df):\n",
    "    return df.quantile(0.25).round(1)\n",
    "\n",
    "def mean(df):\n",
    "    return df.mean().round(1)\n",
    "\n",
    "def median(df):\n",
    "    return df.quantile(0.5).round(1)\n",
    "\n",
    "def upper_quartile(df):\n",
    "    return df.quantile(0.75).round(1)\n",
    "\n",
    "num_col_info_df = train.select_dtypes(include=np.number)\n",
    "num_col_info_df = num_col_info_df.agg([missing_ratio, \"min\", mean, lower_quartile, median, upper_quartile, \"max\"])\n",
    "num_col_info_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0309316",
   "metadata": {
    "papermill": {
     "duration": 0.026495,
     "end_time": "2024-05-15T17:39:08.216333",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.189838",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Với các cột categorical, các giá trị được phân bố như thế nào?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de24283",
   "metadata": {
    "papermill": {
     "duration": 0.028526,
     "end_time": "2024-05-15T17:39:08.271607",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.243081",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Ở phần này, với các cột có kiểu dữ liệu `categorical`, ta sẽ tính:\n",
    "+ Tỉ lệ phần trăm các giá trị thiếu.\n",
    "+ Số lượng các giá trị khác nhau (ta sẽ không tính các giá trị thiếu).\n",
    "+ Tỉ lệ phần trăm của mỗi giá trị trong cột."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "85145f5f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:08.333111Z",
     "iopub.status.busy": "2024-05-15T17:39:08.332744Z",
     "iopub.status.idle": "2024-05-15T17:39:08.364699Z",
     "shell.execute_reply": "2024-05-15T17:39:08.363815Z"
    },
    "papermill": {
     "duration": 0.065633,
     "end_time": "2024-05-15T17:39:08.366820",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.301187",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dependency</th>\n",
       "      <th>edjefe</th>\n",
       "      <th>edjefa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>missing_ratio</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_values</th>\n",
       "      <td>31</td>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>value_ratios</th>\n",
       "      <td>{'yes': 22.9, 'no': 18.3, '.5': 15.7, '2': 7.6...</td>\n",
       "      <td>{'no': 39.4, '6': 19.3, '11': 7.9, '9': 5.1, '...</td>\n",
       "      <td>{'no': 65.2, '6': 9.9, '11': 4.2, '9': 2.5, '8...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      dependency  \\\n",
       "missing_ratio                                                0.0   \n",
       "num_values                                                    31   \n",
       "value_ratios   {'yes': 22.9, 'no': 18.3, '.5': 15.7, '2': 7.6...   \n",
       "\n",
       "                                                          edjefe  \\\n",
       "missing_ratio                                                0.0   \n",
       "num_values                                                    22   \n",
       "value_ratios   {'no': 39.4, '6': 19.3, '11': 7.9, '9': 5.1, '...   \n",
       "\n",
       "                                                          edjefa  \n",
       "missing_ratio                                                0.0  \n",
       "num_values                                                    22  \n",
       "value_ratios   {'no': 65.2, '6': 9.9, '11': 4.2, '9': 2.5, '8...  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def num_values(df):\n",
    "    return df.nunique()\n",
    "\n",
    "def value_ratios(df):\n",
    "    value_counts = df.value_counts(normalize=True)*100\n",
    "    value_ratios = value_counts.round(1).to_dict()\n",
    "    value_ratios = dict(sorted(value_ratios.items(), key=lambda item: item[1], reverse=True))\n",
    "    return value_ratios\n",
    "\n",
    "cat_col_info_df = train.select_dtypes(include='object').drop(columns=['Id','idhogar'])\n",
    "cat_col_info_df = cat_col_info_df.agg([missing_ratio, num_values, value_ratios])\n",
    "cat_col_info_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5052aa87",
   "metadata": {
    "papermill": {
     "duration": 0.028555,
     "end_time": "2024-05-15T17:39:08.422594",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.394039",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Nhóm nhận thấy ý nghĩa của các cột `dependency`, `edjefe`, `edjefa` nên có dữ liệu số. Ta sẽ giải quyết vấn đề này theo phương pháp sau:\n",
    "+ Đối với cột `dependency`, ta sẽ tính toán lại bằng cách lấy căn bậc 2 từ cột `SQBdependency`.\n",
    "+ Đối với 2 biến trình độ giáo dục `edjefe` và `edjefa`, ta sẽ chuẩn hóa **yes = 1**, **no = 0** và với các giá trị còn lại ta sẽ chuyển sang dữ liệu số."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e9056d60",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:08.481561Z",
     "iopub.status.busy": "2024-05-15T17:39:08.481086Z",
     "iopub.status.idle": "2024-05-15T17:39:08.509268Z",
     "shell.execute_reply": "2024-05-15T17:39:08.507818Z"
    },
    "papermill": {
     "duration": 0.059686,
     "end_time": "2024-05-15T17:39:08.511849",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.452163",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "mapping = {\"yes\": 1, \"no\": 0}\n",
    "\n",
    "for df in [train, test]:\n",
    "    df['dependency'] = np.sqrt(df['SQBdependency'])\n",
    "    df['edjefa'] = df['edjefa'].replace(mapping).astype(np.float64)\n",
    "    df['edjefe'] = df['edjefe'].replace(mapping).astype(np.float64)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "534573b9",
   "metadata": {
    "papermill": {
     "duration": 0.026439,
     "end_time": "2024-05-15T17:39:08.565039",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.538600",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Xử lý các giá trị thiếu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "84e3513a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:08.622632Z",
     "iopub.status.busy": "2024-05-15T17:39:08.621696Z",
     "iopub.status.idle": "2024-05-15T17:39:08.636038Z",
     "shell.execute_reply": "2024-05-15T17:39:08.634968Z"
    },
    "papermill": {
     "duration": 0.045971,
     "end_time": "2024-05-15T17:39:08.638636",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.592665",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "v2a1         6860\n",
       "v18q1        7342\n",
       "rez_esc      7928\n",
       "meaneduc        5\n",
       "SQBmeaned       5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.isnull().sum()[train.isnull().sum() > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ef0067f",
   "metadata": {
    "papermill": {
     "duration": 0.026413,
     "end_time": "2024-05-15T17:39:08.691986",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.665573",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Cột v2a1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbda23b6",
   "metadata": {
    "papermill": {
     "duration": 0.029453,
     "end_time": "2024-05-15T17:39:08.748819",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.719366",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "`v2a1` là số tiền thuê nhà hằng tháng của hộ gia đình đó. Nhóm giả định rằng trong số những hộ có giá trị `v2a1` bị thiếu tức là hộ gia đình đó có khả năng đã sở hữu căn nhà."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "243c885f",
   "metadata": {
    "papermill": {
     "duration": 0.027519,
     "end_time": "2024-05-15T17:39:08.803835",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.776316",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Do đó, nếu `tipovivi1` = 1 thì ta sẽ điền vào `v2a1` con trống là giá trị 0. Đồng nghĩa là hộ gia đình đó không trả thanh toán tiền thuê hằng tháng khi đã sở hữu căn nhà."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bea42ce6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:08.861729Z",
     "iopub.status.busy": "2024-05-15T17:39:08.861101Z",
     "iopub.status.idle": "2024-05-15T17:39:08.870209Z",
     "shell.execute_reply": "2024-05-15T17:39:08.869069Z"
    },
    "papermill": {
     "duration": 0.041157,
     "end_time": "2024-05-15T17:39:08.872724",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.831567",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train.loc[(pd.isnull(train['v2a1']) & train['tipovivi1'] == 1), 'v2a1'] = 0\n",
    "test.loc[(pd.isnull(test['v2a1']) & test['tipovivi1'] == 1), 'v2a1'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0c32ff6c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:08.932246Z",
     "iopub.status.busy": "2024-05-15T17:39:08.931169Z",
     "iopub.status.idle": "2024-05-15T17:39:08.943248Z",
     "shell.execute_reply": "2024-05-15T17:39:08.942025Z"
    },
    "papermill": {
     "duration": 0.04348,
     "end_time": "2024-05-15T17:39:08.945749",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.902269",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Series([], Name: count, dtype: int64)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[(train['tipovivi4'] == 1) | (train['tipovivi5'] == 1)]['v2a1'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "78143538",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:09.005307Z",
     "iopub.status.busy": "2024-05-15T17:39:09.004884Z",
     "iopub.status.idle": "2024-05-15T17:39:09.018074Z",
     "shell.execute_reply": "2024-05-15T17:39:09.016389Z"
    },
    "papermill": {
     "duration": 0.046667,
     "end_time": "2024-05-15T17:39:09.020682",
     "exception": false,
     "start_time": "2024-05-15T17:39:08.974015",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train.loc[(pd.isnull(train['v2a1']) & train['tipovivi4'] == 1), 'v2a1'] = 0\n",
    "train.loc[(pd.isnull(train['v2a1']) & train['tipovivi5'] == 1), 'v2a1'] = 0\n",
    "\n",
    "test.loc[(pd.isnull(test['v2a1']) & test['tipovivi4'] == 1), 'v2a1'] = 0\n",
    "test.loc[(pd.isnull(test['v2a1']) & test['tipovivi5'] == 1), 'v2a1'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1c783e0",
   "metadata": {
    "papermill": {
     "duration": 0.031418,
     "end_time": "2024-05-15T17:39:09.079960",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.048542",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Cột v18q1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72ea7c67",
   "metadata": {
    "papermill": {
     "duration": 0.026489,
     "end_time": "2024-05-15T17:39:09.133437",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.106948",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "`v18q1` là số lượng máy tính bảng mà hộ giá định sở hữu. Nhóm thực hiện kiểm tra số lượng được thống kê trong cột này."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2c116e35",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:09.191653Z",
     "iopub.status.busy": "2024-05-15T17:39:09.191256Z",
     "iopub.status.idle": "2024-05-15T17:39:09.200489Z",
     "shell.execute_reply": "2024-05-15T17:39:09.199375Z"
    },
    "papermill": {
     "duration": 0.04198,
     "end_time": "2024-05-15T17:39:09.202728",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.160748",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "v18q1\n",
       "1.0    1586\n",
       "2.0     444\n",
       "3.0     129\n",
       "4.0      37\n",
       "5.0      13\n",
       "6.0       6\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['v18q1'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cda1b37b",
   "metadata": {
    "papermill": {
     "duration": 0.027099,
     "end_time": "2024-05-15T17:39:09.258607",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.231508",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Số liệu đã thống kê hoàn toàn không có giá trị 0, ta có thể đảm bảo rằng những giá trị thiếu có nghĩa là hộ gia đình đó không sở hữu máy tính bảng."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ffd806c6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:09.318021Z",
     "iopub.status.busy": "2024-05-15T17:39:09.317541Z",
     "iopub.status.idle": "2024-05-15T17:39:09.324237Z",
     "shell.execute_reply": "2024-05-15T17:39:09.323104Z"
    },
    "papermill": {
     "duration": 0.039285,
     "end_time": "2024-05-15T17:39:09.326802",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.287517",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train['v18q1'].fillna(0, inplace=True)\n",
    "test['v18q1'].fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adb37ec1",
   "metadata": {
    "papermill": {
     "duration": 0.027087,
     "end_time": "2024-05-15T17:39:09.381509",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.354422",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Cột rez_esc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "852013d2",
   "metadata": {
    "papermill": {
     "duration": 0.028252,
     "end_time": "2024-05-15T17:39:09.440101",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.411849",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "`rez_esc` là số năm đi học trễ. Các hộ gia đình có giá trị trống có thể không có trẻ em hiện đang đi học. Ta đi kiểm tra tuổi của những người không có giá trị bị thiếu trong cột này."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "93f1661e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:09.497968Z",
     "iopub.status.busy": "2024-05-15T17:39:09.497568Z",
     "iopub.status.idle": "2024-05-15T17:39:09.505136Z",
     "shell.execute_reply": "2024-05-15T17:39:09.504226Z"
    },
    "papermill": {
     "duration": 0.03971,
     "end_time": "2024-05-15T17:39:09.507780",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.468070",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "rez_esc\n",
       "0.0    1211\n",
       "1.0     227\n",
       "2.0      98\n",
       "3.0      55\n",
       "4.0      29\n",
       "5.0       9\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['rez_esc'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "145a8858",
   "metadata": {
    "papermill": {
     "duration": 0.026893,
     "end_time": "2024-05-15T17:39:09.563801",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.536908",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Sau đọc thảo luận về dữ liệu cuộc thi này, nhóm nhận thấy rằng thuộc tính `rez_esc` chỉ được xác định cho các đối tượng trong độ tuổi từ 7 đến 19. Do đó, các đối tượng nhỏ hơn 7 tuổi hoặc từ 19 tuổi trở lên thì giá trị `rez_esc` sẽ được đặt là 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8f95f79e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:09.626211Z",
     "iopub.status.busy": "2024-05-15T17:39:09.625851Z",
     "iopub.status.idle": "2024-05-15T17:39:09.634225Z",
     "shell.execute_reply": "2024-05-15T17:39:09.633087Z"
    },
    "papermill": {
     "duration": 0.041748,
     "end_time": "2024-05-15T17:39:09.636970",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.595222",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train.loc[((train['age'] > 19) | (train['age'] < 7)) & (train['rez_esc'].isnull()), 'rez_esc'] = 0\n",
    "test.loc[((test['age'] > 19) | (test['age'] < 7)) & (test['rez_esc'].isnull()), 'rez_esc'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9388e68",
   "metadata": {
    "papermill": {
     "duration": 0.029223,
     "end_time": "2024-05-15T17:39:09.693636",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.664413",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Đối với những giá trị còn thiếu với đối tượng có độ tuổi từ 7 đến 19 sẽ điền và giá trị mode của cột `rez_esc`. Ngoài ra, theo phần thảo luận cuộc thi thì giá trị của `rez_esc` sẽ nằm trong đoạn **1 - 5** nên ta sẽ chuyển những giá trị lớn hơn 5 thành 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "81ca44b7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:09.755267Z",
     "iopub.status.busy": "2024-05-15T17:39:09.754154Z",
     "iopub.status.idle": "2024-05-15T17:39:09.797649Z",
     "shell.execute_reply": "2024-05-15T17:39:09.796773Z"
    },
    "papermill": {
     "duration": 0.078808,
     "end_time": "2024-05-15T17:39:09.800123",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.721315",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "for df in [train, test]:\n",
    "    for i in range(7, 20):\n",
    "        mode_value = df.loc[df['age'] == i, 'rez_esc'].mode()\n",
    "        if mode_value.empty: \n",
    "            mode_value=0\n",
    "        else:\n",
    "            mode_value=mode_value[0]\n",
    "            \n",
    "        df.loc[(df['age'] == i) & (df['rez_esc'].isna()), 'rez_esc'] = mode_value\n",
    "\n",
    "    df.loc[df['rez_esc'] > 5, 'rez_esc'] = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81ea58cf",
   "metadata": {
    "papermill": {
     "duration": 0.030989,
     "end_time": "2024-05-15T17:39:09.859851",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.828862",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Cột meaneduc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31d5875a",
   "metadata": {
    "papermill": {
     "duration": 0.028338,
     "end_time": "2024-05-15T17:39:09.916226",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.887888",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "`meaneduc` là số năm học trung bình của người lớn (hơn 18 tuổi)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8157ec38",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:09.975387Z",
     "iopub.status.busy": "2024-05-15T17:39:09.974793Z",
     "iopub.status.idle": "2024-05-15T17:39:09.996551Z",
     "shell.execute_reply": "2024-05-15T17:39:09.995188Z"
    },
    "papermill": {
     "duration": 0.054963,
     "end_time": "2024-05-15T17:39:09.999069",
     "exception": false,
     "start_time": "2024-05-15T17:39:09.944106",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>v2a1</th>\n",
       "      <th>hacdor</th>\n",
       "      <th>rooms</th>\n",
       "      <th>hacapo</th>\n",
       "      <th>v14a</th>\n",
       "      <th>refrig</th>\n",
       "      <th>v18q</th>\n",
       "      <th>v18q1</th>\n",
       "      <th>r4h1</th>\n",
       "      <th>...</th>\n",
       "      <th>SQBescolari</th>\n",
       "      <th>SQBage</th>\n",
       "      <th>SQBhogar_total</th>\n",
       "      <th>SQBedjefe</th>\n",
       "      <th>SQBhogar_nin</th>\n",
       "      <th>SQBovercrowding</th>\n",
       "      <th>SQBdependency</th>\n",
       "      <th>SQBmeaned</th>\n",
       "      <th>agesq</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1291</th>\n",
       "      <td>ID_bd8e11b0f</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>100</td>\n",
       "      <td>324</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.04</td>\n",
       "      <td>64.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>324</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840</th>\n",
       "      <td>ID_46ff87316</td>\n",
       "      <td>110000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>36</td>\n",
       "      <td>324</td>\n",
       "      <td>4</td>\n",
       "      <td>16</td>\n",
       "      <td>4</td>\n",
       "      <td>4.00</td>\n",
       "      <td>64.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>324</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1841</th>\n",
       "      <td>ID_69f50bf3e</td>\n",
       "      <td>110000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>16</td>\n",
       "      <td>324</td>\n",
       "      <td>4</td>\n",
       "      <td>16</td>\n",
       "      <td>4</td>\n",
       "      <td>4.00</td>\n",
       "      <td>64.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>324</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2049</th>\n",
       "      <td>ID_db3168f9f</td>\n",
       "      <td>180000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>144</td>\n",
       "      <td>361</td>\n",
       "      <td>4</td>\n",
       "      <td>144</td>\n",
       "      <td>4</td>\n",
       "      <td>1.00</td>\n",
       "      <td>64.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>361</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2050</th>\n",
       "      <td>ID_2a7615902</td>\n",
       "      <td>180000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>144</td>\n",
       "      <td>361</td>\n",
       "      <td>4</td>\n",
       "      <td>144</td>\n",
       "      <td>4</td>\n",
       "      <td>1.00</td>\n",
       "      <td>64.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>361</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 143 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Id      v2a1  hacdor  rooms  hacapo  v14a  refrig  v18q  \\\n",
       "1291  ID_bd8e11b0f       0.0       0      7       0     1       1     0   \n",
       "1840  ID_46ff87316  110000.0       0      1       0     1       1     0   \n",
       "1841  ID_69f50bf3e  110000.0       0      1       0     1       1     0   \n",
       "2049  ID_db3168f9f  180000.0       0      3       0     1       1     0   \n",
       "2050  ID_2a7615902  180000.0       0      3       0     1       1     0   \n",
       "\n",
       "      v18q1  r4h1  ...  SQBescolari  SQBage  SQBhogar_total  SQBedjefe  \\\n",
       "1291    0.0     0  ...          100     324               1          0   \n",
       "1840    0.0     0  ...           36     324               4         16   \n",
       "1841    0.0     0  ...           16     324               4         16   \n",
       "2049    0.0     0  ...          144     361               4        144   \n",
       "2050    0.0     0  ...          144     361               4        144   \n",
       "\n",
       "      SQBhogar_nin  SQBovercrowding  SQBdependency  SQBmeaned  agesq  Target  \n",
       "1291             1             0.04           64.0        NaN    324       4  \n",
       "1840             4             4.00           64.0        NaN    324       4  \n",
       "1841             4             4.00           64.0        NaN    324       4  \n",
       "2049             4             1.00           64.0        NaN    361       4  \n",
       "2050             4             1.00           64.0        NaN    361       4  \n",
       "\n",
       "[5 rows x 143 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.loc[pd.isnull(train['meaneduc'])].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f93f73",
   "metadata": {
    "papermill": {
     "duration": 0.029358,
     "end_time": "2024-05-15T17:39:10.056238",
     "exception": false,
     "start_time": "2024-05-15T17:39:10.026880",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Nhóm sẽ điền các giá trị thiếu bằng cách đi tính giá trị trung bình của số năm học `escolari` theo từng hộ gia đình."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2280ec9d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:10.117938Z",
     "iopub.status.busy": "2024-05-15T17:39:10.117556Z",
     "iopub.status.idle": "2024-05-15T17:39:10.715612Z",
     "shell.execute_reply": "2024-05-15T17:39:10.714146Z"
    },
    "papermill": {
     "duration": 0.633151,
     "end_time": "2024-05-15T17:39:10.718570",
     "exception": false,
     "start_time": "2024-05-15T17:39:10.085419",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "mean_escolari_by_idhogar = train.groupby('idhogar')['escolari'].mean()\n",
    "train['meaneduc'] = train.apply(lambda row: int(mean_escolari_by_idhogar[row['idhogar']]) if pd.isnull(row['meaneduc']) else row['meaneduc'], axis=1)\n",
    "\n",
    "mean_escolari_by_idhogar = test.groupby('idhogar')['escolari'].mean()\n",
    "test['meaneduc'] = test.apply(lambda row: int(mean_escolari_by_idhogar[row['idhogar']]) if pd.isnull(row['meaneduc']) else row['meaneduc'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d627b382",
   "metadata": {
    "papermill": {
     "duration": 0.028696,
     "end_time": "2024-05-15T17:39:10.778494",
     "exception": false,
     "start_time": "2024-05-15T17:39:10.749798",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Cột SQBmeaned"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ded53b6",
   "metadata": {
    "papermill": {
     "duration": 0.02681,
     "end_time": "2024-05-15T17:39:10.833089",
     "exception": false,
     "start_time": "2024-05-15T17:39:10.806279",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "`SQBmeaned` là bình phương của `meaneduc` nên với những giá trị thiếu ở cột này nhóm sẽ đi tính lại giá trị bình phương và điền vào."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fb216554",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:10.892955Z",
     "iopub.status.busy": "2024-05-15T17:39:10.891752Z",
     "iopub.status.idle": "2024-05-15T17:39:10.902090Z",
     "shell.execute_reply": "2024-05-15T17:39:10.901301Z"
    },
    "papermill": {
     "duration": 0.04292,
     "end_time": "2024-05-15T17:39:10.904674",
     "exception": false,
     "start_time": "2024-05-15T17:39:10.861754",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train.loc[pd.isnull(train['SQBmeaned']), 'SQBmeaned'] = train.loc[pd.isnull(train['SQBmeaned']), 'meaneduc'] ** 2\n",
    "test.loc[pd.isnull(test['SQBmeaned']), 'SQBmeaned'] = test.loc[pd.isnull(test['SQBmeaned']), 'meaneduc'] ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d96729cd",
   "metadata": {
    "papermill": {
     "duration": 0.032967,
     "end_time": "2024-05-15T17:39:10.969438",
     "exception": false,
     "start_time": "2024-05-15T17:39:10.936471",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Xây dựng mô hình học máy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35bd3717",
   "metadata": {
    "papermill": {
     "duration": 0.030063,
     "end_time": "2024-05-15T17:39:11.030582",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.000519",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Tổng hợp dữ liệu để xử lý trên các thuộc tính."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "356cac35",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:11.091414Z",
     "iopub.status.busy": "2024-05-15T17:39:11.090981Z",
     "iopub.status.idle": "2024-05-15T17:39:11.163375Z",
     "shell.execute_reply": "2024-05-15T17:39:11.162218Z"
    },
    "papermill": {
     "duration": 0.105409,
     "end_time": "2024-05-15T17:39:11.166433",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.061024",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ntrain = train.shape[0]\n",
    "ntest = test.shape[0]\n",
    "\n",
    "all_data = pd.concat((train, test)).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0567fcc",
   "metadata": {
    "papermill": {
     "duration": 0.030349,
     "end_time": "2024-05-15T17:39:11.226244",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.195895",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Loại bỏ các cột chỉ có 1 giá trị độc nhất"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9cd6bba",
   "metadata": {
    "papermill": {
     "duration": 0.034902,
     "end_time": "2024-05-15T17:39:11.293441",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.258539",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Đối với các cột chỉ tồn tại 1 giá trị duy nhất, việc xuất hiện của nó có thể làm cho quá trình dự đoán kết quả trở nên cồng kềnh hơn một cách không cần thiết."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "d4181936",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:11.355226Z",
     "iopub.status.busy": "2024-05-15T17:39:11.354811Z",
     "iopub.status.idle": "2024-05-15T17:39:11.414403Z",
     "shell.execute_reply": "2024-05-15T17:39:11.413251Z"
    },
    "papermill": {
     "duration": 0.094779,
     "end_time": "2024-05-15T17:39:11.417109",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.322330",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "for column in all_data.columns:\n",
    "    if all_data[column].nunique() == 1:\n",
    "        all_data.drop(column, axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a589f453",
   "metadata": {
    "papermill": {
     "duration": 0.028425,
     "end_time": "2024-05-15T17:39:11.474729",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.446304",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Loại bỏ những thuộc tính dư thừa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99fc08d7",
   "metadata": {
    "papermill": {
     "duration": 0.027197,
     "end_time": "2024-05-15T17:39:11.529098",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.501901",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Tạo biến ordinal từ dữ liệu đã được one-hot encode\n",
    "\n",
    "Các thuộc tính như `epared`, `etecho`, `eviv` có thể được chuyển về dạng dữ liệu ordinal với quy ước **(bad, regular, good) -> (0, 1, 2)** và`instlevel` với giá trị từ **1 - 9**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "8c133715",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:11.586246Z",
     "iopub.status.busy": "2024-05-15T17:39:11.585872Z",
     "iopub.status.idle": "2024-05-15T17:39:11.592148Z",
     "shell.execute_reply": "2024-05-15T17:39:11.590918Z"
    },
    "papermill": {
     "duration": 0.038477,
     "end_time": "2024-05-15T17:39:11.594680",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.556203",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_numeric(data, status_name):\n",
    "    status_cols = [s for s in data.columns.tolist() if status_name in s]\n",
    "    status_df = data[status_cols]\n",
    "    status_df.columns = list(range(status_df.shape[1]))\n",
    "    status_numeric = status_df.idxmax(1)\n",
    "    status_numeric.name = status_name\n",
    "    data = pd.concat([data, status_numeric], axis=1)\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "aa54dfa8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:11.653140Z",
     "iopub.status.busy": "2024-05-15T17:39:11.652683Z",
     "iopub.status.idle": "2024-05-15T17:39:11.831158Z",
     "shell.execute_reply": "2024-05-15T17:39:11.830002Z"
    },
    "papermill": {
     "duration": 0.210974,
     "end_time": "2024-05-15T17:39:11.833878",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.622904",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "status_name_list = ['epared', 'etecho', 'eviv', 'instlevel']\n",
    "for status_name in status_name_list:\n",
    "    all_data = get_numeric(all_data, status_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee3291fa",
   "metadata": {
    "papermill": {
     "duration": 0.027233,
     "end_time": "2024-05-15T17:39:11.889849",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.862616",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Xóa những thuộc tính không cần thiết"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2104eaf8",
   "metadata": {
    "papermill": {
     "duration": 0.027164,
     "end_time": "2024-05-15T17:39:11.944852",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.917688",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Nhóm nhận thấy có những thuộc tính có thể được xác định bằng những thuộc tính khác trong dữ liệu.\n",
    "\n",
    "- Nhóm thuộc tính sau cùng ý nghĩa với `r4h` và `r4m`:\n",
    "    ```\n",
    "    r4t1, persons younger than 12 years of age\n",
    "    r4t2, persons 12 years of age and older\n",
    "    r4t3, Total persons in the household\n",
    "    ```\n",
    "\n",
    "- Các thuộc tính sau mang cùng ý nghĩa với `hogar_total`:\n",
    "    ```\n",
    "    tamhog, size of the household\n",
    "    tamviv, number of persons living in the household\n",
    "    hhsize, household size\n",
    "    r4t3, Total persons in the household\n",
    "    ```\n",
    "\n",
    "- `v18q` có thể được tạo ra từ `v18q1`.\n",
    "- `mobilephone` có thể được tạo ra từ `qmobilephone`.\n",
    "- `area2` có thể suy ra từ `area1`.\n",
    "- `female` có thể suy ra từ `male`.\n",
    "- `epared1~3`, `etecho1~3`, `eviv1~3`, `instlevel1~9` do đã được chuyển đổi thành dữ liệu ordinal nên sẽ không dùng đến nữa.\n",
    "- `SQBage` có trùng dữ liệu với `agesq`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "ca7812b6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:12.002846Z",
     "iopub.status.busy": "2024-05-15T17:39:12.002469Z",
     "iopub.status.idle": "2024-05-15T17:39:12.018792Z",
     "shell.execute_reply": "2024-05-15T17:39:12.017575Z"
    },
    "papermill": {
     "duration": 0.048477,
     "end_time": "2024-05-15T17:39:12.021706",
     "exception": false,
     "start_time": "2024-05-15T17:39:11.973229",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "redundant_features = ['r4t1', 'r4t2', 'r4t3', \n",
    "                      'tamhog', 'tamviv', 'hhsize',\n",
    "                      'v18q', 'mobilephone', 'area2', 'female', 'SQBage',\n",
    "                      'epared1', 'epared2', 'epared3', \n",
    "                      'etecho1', 'etecho2', 'etecho3',\n",
    "                      'eviv1', 'eviv2', 'eviv3', \n",
    "                      'instlevel1', 'instlevel2', 'instlevel3', 'instlevel4', 'instlevel5', 'instlevel6', 'instlevel7', 'instlevel8', 'instlevel9']\n",
    "\n",
    "all_data.drop(columns=redundant_features, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5abbe7f",
   "metadata": {
    "papermill": {
     "duration": 0.030409,
     "end_time": "2024-05-15T17:39:12.079787",
     "exception": false,
     "start_time": "2024-05-15T17:39:12.049378",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Ngoài ra, nhóm sẽ không loại bỏ các biến bình phương. Vì các biến dữ liệu này thường được biến đổi như một phần của Feature Engineering vì nó có thể giúp các mô hình tuyến tính tìm hiểu các mối quan hệ phi tuyến tính."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5a4f3377",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:12.139592Z",
     "iopub.status.busy": "2024-05-15T17:39:12.138975Z",
     "iopub.status.idle": "2024-05-15T17:39:12.161518Z",
     "shell.execute_reply": "2024-05-15T17:39:12.160593Z"
    },
    "papermill": {
     "duration": 0.055069,
     "end_time": "2024-05-15T17:39:12.164577",
     "exception": false,
     "start_time": "2024-05-15T17:39:12.109508",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Target    23856\n",
       "dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data.isnull().sum()[all_data.isnull().sum() > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91b981c3",
   "metadata": {
    "papermill": {
     "duration": 0.028742,
     "end_time": "2024-05-15T17:39:12.222152",
     "exception": false,
     "start_time": "2024-05-15T17:39:12.193410",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Trích lọc đặc trưng bằng thông số thống kê"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36145ad4",
   "metadata": {
    "papermill": {
     "duration": 0.027157,
     "end_time": "2024-05-15T17:39:12.276707",
     "exception": false,
     "start_time": "2024-05-15T17:39:12.249550",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Để kết hợp dữ liệu của từng cá nhân vào dữ liệu của cả hộ gia đình, ta cần tổng hợp dữ liệu đó cho từng hộ gia đình. Cách đơn giản nhất để thực hiện việc này là nhóm dữ liệu theo `idhogar` rồi tổng hợp dữ liệu. Tuy nhiên, các dữ liệu **boolean** có thể giống nhau, và sẽ tạo ra nhiều cột dư thừa mà sau đó chúng ta sẽ cần phải loại bỏ sau khi triển khai."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "55267a12",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:12.335007Z",
     "iopub.status.busy": "2024-05-15T17:39:12.334632Z",
     "iopub.status.idle": "2024-05-15T17:39:12.340302Z",
     "shell.execute_reply": "2024-05-15T17:39:12.339063Z"
    },
    "papermill": {
     "duration": 0.036851,
     "end_time": "2024-05-15T17:39:12.342851",
     "exception": false,
     "start_time": "2024-05-15T17:39:12.306000",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ind_bool = ['dis', 'male',\n",
    "            'estadocivil1', 'estadocivil2', 'estadocivil3', 'estadocivil4', 'estadocivil5', 'estadocivil6', 'estadocivil7', \n",
    "            'parentesco1', 'parentesco2',  'parentesco3', 'parentesco4', 'parentesco5', 'parentesco6', \n",
    "            'parentesco7', 'parentesco8',  'parentesco9', 'parentesco10', 'parentesco11', 'parentesco12']\n",
    "\n",
    "ind_ordered = ['escolari', 'age', 'instlevel']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "d62f6cac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:12.400900Z",
     "iopub.status.busy": "2024-05-15T17:39:12.400513Z",
     "iopub.status.idle": "2024-05-15T17:39:30.754884Z",
     "shell.execute_reply": "2024-05-15T17:39:30.753623Z"
    },
    "papermill": {
     "duration": 18.386534,
     "end_time": "2024-05-15T17:39:30.757755",
     "exception": false,
     "start_time": "2024-05-15T17:39:12.371221",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>escolari-mean</th>\n",
       "      <th>escolari-max</th>\n",
       "      <th>escolari-min</th>\n",
       "      <th>escolari-sum</th>\n",
       "      <th>escolari-std_</th>\n",
       "      <th>age-mean</th>\n",
       "      <th>age-max</th>\n",
       "      <th>age-min</th>\n",
       "      <th>age-sum</th>\n",
       "      <th>age-std_</th>\n",
       "      <th>...</th>\n",
       "      <th>parentesco11-mean</th>\n",
       "      <th>parentesco11-max</th>\n",
       "      <th>parentesco11-min</th>\n",
       "      <th>parentesco11-sum</th>\n",
       "      <th>parentesco11-std_</th>\n",
       "      <th>parentesco12-mean</th>\n",
       "      <th>parentesco12-max</th>\n",
       "      <th>parentesco12-min</th>\n",
       "      <th>parentesco12-sum</th>\n",
       "      <th>parentesco12-std_</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>idhogar</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>000a08204</th>\n",
       "      <td>8.666667</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>26</td>\n",
       "      <td>6.182412</td>\n",
       "      <td>20.666667</td>\n",
       "      <td>30</td>\n",
       "      <td>4</td>\n",
       "      <td>62</td>\n",
       "      <td>11.813363</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>000bce7c4</th>\n",
       "      <td>2.500000</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>61.500000</td>\n",
       "      <td>63</td>\n",
       "      <td>60</td>\n",
       "      <td>123</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>001845fb0</th>\n",
       "      <td>10.250000</td>\n",
       "      <td>14</td>\n",
       "      <td>6</td>\n",
       "      <td>41</td>\n",
       "      <td>2.861381</td>\n",
       "      <td>35.500000</td>\n",
       "      <td>52</td>\n",
       "      <td>19</td>\n",
       "      <td>142</td>\n",
       "      <td>14.221463</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>001ff74ca</th>\n",
       "      <td>8.000000</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>003123ec2</th>\n",
       "      <td>3.250000</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>3.269174</td>\n",
       "      <td>12.750000</td>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>10.779031</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 120 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           escolari-mean  escolari-max  escolari-min  escolari-sum  \\\n",
       "idhogar                                                              \n",
       "000a08204       8.666667            14             0            26   \n",
       "000bce7c4       2.500000             5             0             5   \n",
       "001845fb0      10.250000            14             6            41   \n",
       "001ff74ca       8.000000            16             0            16   \n",
       "003123ec2       3.250000             7             0            13   \n",
       "\n",
       "           escolari-std_   age-mean  age-max  age-min  age-sum   age-std_  \\\n",
       "idhogar                                                                     \n",
       "000a08204       6.182412  20.666667       30        4       62  11.813363   \n",
       "000bce7c4       2.500000  61.500000       63       60      123   1.500000   \n",
       "001845fb0       2.861381  35.500000       52       19      142  14.221463   \n",
       "001ff74ca       8.000000  19.000000       38        0       38  19.000000   \n",
       "003123ec2       3.269174  12.750000       24        1       51  10.779031   \n",
       "\n",
       "           ...  parentesco11-mean  parentesco11-max  parentesco11-min  \\\n",
       "idhogar    ...                                                          \n",
       "000a08204  ...                0.0                 0                 0   \n",
       "000bce7c4  ...                0.0                 0                 0   \n",
       "001845fb0  ...                0.0                 0                 0   \n",
       "001ff74ca  ...                0.0                 0                 0   \n",
       "003123ec2  ...                0.0                 0                 0   \n",
       "\n",
       "           parentesco11-sum  parentesco11-std_  parentesco12-mean  \\\n",
       "idhogar                                                             \n",
       "000a08204                 0                0.0                0.0   \n",
       "000bce7c4                 0                0.0                0.0   \n",
       "001845fb0                 0                0.0                0.0   \n",
       "001ff74ca                 0                0.0                0.0   \n",
       "003123ec2                 0                0.0                0.0   \n",
       "\n",
       "           parentesco12-max  parentesco12-min  parentesco12-sum  \\\n",
       "idhogar                                                           \n",
       "000a08204                 0                 0                 0   \n",
       "000bce7c4                 0                 0                 0   \n",
       "001845fb0                 0                 0                 0   \n",
       "001ff74ca                 0                 0                 0   \n",
       "003123ec2                 0                 0                 0   \n",
       "\n",
       "           parentesco12-std_  \n",
       "idhogar                       \n",
       "000a08204                0.0  \n",
       "000bce7c4                0.0  \n",
       "001845fb0                0.0  \n",
       "001ff74ca                0.0  \n",
       "003123ec2                0.0  \n",
       "\n",
       "[5 rows x 120 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f = lambda x: x.std(ddof=0)\n",
    "f.__name__ = 'std_'\n",
    "ind_agg = all_data.groupby('idhogar')[ind_ordered + ind_bool].agg(['mean', 'max', 'min', 'sum', f])\n",
    "\n",
    "new_cols = []\n",
    "for col in ind_agg.columns.levels[0]:\n",
    "    for stat in ind_agg.columns.levels[1]:\n",
    "        new_cols.append(f'{col}-{stat}')\n",
    "\n",
    "ind_agg.columns = new_cols\n",
    "ind_agg.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4fd0de21",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:30.818404Z",
     "iopub.status.busy": "2024-05-15T17:39:30.817913Z",
     "iopub.status.idle": "2024-05-15T17:39:31.297438Z",
     "shell.execute_reply": "2024-05-15T17:39:31.295884Z"
    },
    "papermill": {
     "duration": 0.512401,
     "end_time": "2024-05-15T17:39:31.300230",
     "exception": false,
     "start_time": "2024-05-15T17:39:30.787829",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 2 correlated columns to remove\n"
     ]
    }
   ],
   "source": [
    "corr_matrix = ind_agg.corr()\n",
    "upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))\n",
    "\n",
    "to_drop = [column for column in upper.columns if any((abs(upper[column]) > 0.95) & (abs(upper[column]) == 1))]\n",
    "print(f'There are {len(to_drop)} correlated columns to remove')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "48330e15",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:31.362477Z",
     "iopub.status.busy": "2024-05-15T17:39:31.361701Z",
     "iopub.status.idle": "2024-05-15T17:39:31.495952Z",
     "shell.execute_reply": "2024-05-15T17:39:31.494643Z"
    },
    "papermill": {
     "duration": 0.169999,
     "end_time": "2024-05-15T17:39:31.498361",
     "exception": false,
     "start_time": "2024-05-15T17:39:31.328362",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features after dropping the individual level features 212\n"
     ]
    }
   ],
   "source": [
    "all_data = all_data.merge(ind_agg, on = 'idhogar', how = 'left')\n",
    "all_data.drop(columns=ind_bool+ind_ordered+to_drop, inplace=True)\n",
    "print('Number of features after dropping the individual level features', all_data.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd238b11",
   "metadata": {
    "papermill": {
     "duration": 0.028476,
     "end_time": "2024-05-15T17:39:31.554789",
     "exception": false,
     "start_time": "2024-05-15T17:39:31.526313",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Tạo thêm cột phụ biểu hiện thêm thông tin của hộ gia đình"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea87de05",
   "metadata": {
    "papermill": {
     "duration": 0.030002,
     "end_time": "2024-05-15T17:39:31.613259",
     "exception": false,
     "start_time": "2024-05-15T17:39:31.583257",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Trong một hộ gia đình, cần biết rằng người trưởng thành có khả năng làm việc là trụ cột của gia đình, trẻ em và người lớn trên 65 tuổi thì không làm việc. Do đó việc quyết định việc xem xét đánh giá tình trạng của một gia đình không phụ thuộc nhiều vào tỉ lệ các nhóm đối tượng trong hộ gia đình. Vì vậy ta sẽ tạo thêm một số cột liên quan đến vấn đề này."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "71240122",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:31.675081Z",
     "iopub.status.busy": "2024-05-15T17:39:31.674073Z",
     "iopub.status.idle": "2024-05-15T17:39:31.692300Z",
     "shell.execute_reply": "2024-05-15T17:39:31.691346Z"
    },
    "papermill": {
     "duration": 0.052863,
     "end_time": "2024-05-15T17:39:31.695687",
     "exception": false,
     "start_time": "2024-05-15T17:39:31.642824",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def extract_features(df):\n",
    "    df['adult_num'] = df['hogar_adul'] - df['hogar_mayor'] # số lượng người trưởng thành còn khả năng làm việc\n",
    "    df['head_is_adult'] = (df['adult_num'] > 0).astype(int) # có người trưởng thành là trụ cột\n",
    "    df['adult_rate'] = df['adult_num'] / df['hogar_total'] # tỉ lệ người trưởng thành\n",
    "    \n",
    "    df['dependency_num'] = df['hogar_nin'] + df['hogar_mayor'] # số người phụ thuộc\n",
    "    df['dependency_rate'] = df['dependency_num'] / df['hogar_total'] # tỉ lệ phụ thuộc\n",
    "    \n",
    "    df['adult_dependency_rate'] = df['adult_num'] / (df['dependency_num'] + 0.1) # tỉ lệ người trưởng thành trên người phụ thuọc\n",
    "    df['children_rate'] = df['hogar_nin'] / df['hogar_total'] # tỉ lệ trẻ em trong gia đình\n",
    "    df['elder_rate'] = df['hogar_mayor'] / df['hogar_total'] # tỉ lệ người già trong gia đình\n",
    "\n",
    "    df['rent_per_person'] = df['v2a1'] / df['hogar_total'] # giá thuê nhà tính trên mỗi người\n",
    "    df['rent_per_adult'] = df['v2a1'] / (df['adult_num'] + 0.1) # gía thuê nhà tính trên người thưởng thành\n",
    "\n",
    "    df['bedroom_per_person'] = df['bedrooms'] / df['hogar_total'] # số lượng người trung bình mỗi phòng\n",
    "    df['bedroom_per_adult'] = df['bedrooms'] / (df['adult_num'] + 0.1) # số lượng người trung bình mỗi phòng cho người trưởng thành\n",
    "    \n",
    "extract_features(all_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6e7d932",
   "metadata": {
    "papermill": {
     "duration": 0.029081,
     "end_time": "2024-05-15T17:39:31.754395",
     "exception": false,
     "start_time": "2024-05-15T17:39:31.725314",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Chuẩn bị dữ liệu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "202de75c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:31.813499Z",
     "iopub.status.busy": "2024-05-15T17:39:31.812832Z",
     "iopub.status.idle": "2024-05-15T17:39:31.817879Z",
     "shell.execute_reply": "2024-05-15T17:39:31.816777Z"
    },
    "papermill": {
     "duration": 0.037344,
     "end_time": "2024-05-15T17:39:31.820275",
     "exception": false,
     "start_time": "2024-05-15T17:39:31.782931",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train = all_data[:ntrain][:]\n",
    "test = all_data[ntrain:][:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "745a6016",
   "metadata": {
    "papermill": {
     "duration": 0.029092,
     "end_time": "2024-05-15T17:39:31.878366",
     "exception": false,
     "start_time": "2024-05-15T17:39:31.849274",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Lấy dữ liệu dùng để huấn luyện model từ tập `train`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "240bd39b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:31.939656Z",
     "iopub.status.busy": "2024-05-15T17:39:31.939246Z",
     "iopub.status.idle": "2024-05-15T17:39:31.950121Z",
     "shell.execute_reply": "2024-05-15T17:39:31.949233Z"
    },
    "papermill": {
     "duration": 0.043966,
     "end_time": "2024-05-15T17:39:31.952604",
     "exception": false,
     "start_time": "2024-05-15T17:39:31.908638",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "X = train.drop(columns=['Id', 'idhogar', 'Target'])\n",
    "y = train.Target.to_numpy().astype('int') - 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42f2c2ff",
   "metadata": {
    "papermill": {
     "duration": 0.031248,
     "end_time": "2024-05-15T17:39:32.015002",
     "exception": false,
     "start_time": "2024-05-15T17:39:31.983754",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Đối với cột có giá trị lớn (từ 100-100000), nhóm sẽ dùng `Standard Scaler` để đưa dữ liệu về phân phối chuẩn.\n",
    "Đối với các cột còn lại, nhóm sẽ dùng `MinMax Scaler` để đưa giá trị về khoảng (0,1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "af046826",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:32.076319Z",
     "iopub.status.busy": "2024-05-15T17:39:32.075358Z",
     "iopub.status.idle": "2024-05-15T17:39:32.080088Z",
     "shell.execute_reply": "2024-05-15T17:39:32.079260Z"
    },
    "papermill": {
     "duration": 0.038216,
     "end_time": "2024-05-15T17:39:32.082728",
     "exception": false,
     "start_time": "2024-05-15T17:39:32.044512",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "col_std = ['v2a1', 'edjefe', 'edjefa', 'meaneduc', 'rent_per_person']\n",
    "\n",
    "col_minmax =  list(set(X.columns) - set(col_std))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b96a74e7",
   "metadata": {
    "papermill": {
     "duration": 0.028665,
     "end_time": "2024-05-15T17:39:32.140756",
     "exception": false,
     "start_time": "2024-05-15T17:39:32.112091",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Lấy dữ liệu dùng để dự đoán từ tập `test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "03fefcd7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:32.198840Z",
     "iopub.status.busy": "2024-05-15T17:39:32.198485Z",
     "iopub.status.idle": "2024-05-15T17:39:32.217066Z",
     "shell.execute_reply": "2024-05-15T17:39:32.215811Z"
    },
    "papermill": {
     "duration": 0.051018,
     "end_time": "2024-05-15T17:39:32.219782",
     "exception": false,
     "start_time": "2024-05-15T17:39:32.168764",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_test_id = list(test.Id)\n",
    "data_test = test.drop(columns=['Id', 'idhogar', 'Target'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f659b7d8",
   "metadata": {
    "papermill": {
     "duration": 0.027522,
     "end_time": "2024-05-15T17:39:32.277743",
     "exception": false,
     "start_time": "2024-05-15T17:39:32.250221",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Chia dữ liệu huấn luyện thành tập train và test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "b7bcb61a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:32.335262Z",
     "iopub.status.busy": "2024-05-15T17:39:32.334901Z",
     "iopub.status.idle": "2024-05-15T17:39:32.656863Z",
     "shell.execute_reply": "2024-05-15T17:39:32.655593Z"
    },
    "papermill": {
     "duration": 0.353612,
     "end_time": "2024-05-15T17:39:32.659307",
     "exception": false,
     "start_time": "2024-05-15T17:39:32.305695",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train,X_test,y_train,y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb7ee21b",
   "metadata": {
    "papermill": {
     "duration": 0.029604,
     "end_time": "2024-05-15T17:39:32.717474",
     "exception": false,
     "start_time": "2024-05-15T17:39:32.687870",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Xây dựng mô hình"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "729d1961",
   "metadata": {
    "papermill": {
     "duration": 0.03052,
     "end_time": "2024-05-15T17:39:32.776674",
     "exception": false,
     "start_time": "2024-05-15T17:39:32.746154",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Light GBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "bd24ad5e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:32.837098Z",
     "iopub.status.busy": "2024-05-15T17:39:32.836685Z",
     "iopub.status.idle": "2024-05-15T17:39:34.005889Z",
     "shell.execute_reply": "2024-05-15T17:39:34.004650Z"
    },
    "papermill": {
     "duration": 1.203047,
     "end_time": "2024-05-15T17:39:34.008918",
     "exception": false,
     "start_time": "2024-05-15T17:39:32.805871",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import lightgbm as lgb\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "8b4ca608",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:34.067645Z",
     "iopub.status.busy": "2024-05-15T17:39:34.066944Z",
     "iopub.status.idle": "2024-05-15T17:39:34.073034Z",
     "shell.execute_reply": "2024-05-15T17:39:34.071842Z"
    },
    "papermill": {
     "duration": 0.038258,
     "end_time": "2024-05-15T17:39:34.075449",
     "exception": false,
     "start_time": "2024-05-15T17:39:34.037191",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_lgbm = lgb.LGBMClassifier(class_weight='balanced', boosting_type='dart',\n",
    "                         drop_rate=0.9, min_data_in_leaf=100, \n",
    "                         max_bin=255,\n",
    "                         n_estimators=500,\n",
    "                         bagging_fraction=0.01,\n",
    "                         min_sum_hessian_in_leaf=1,\n",
    "                         importance_type='gain',\n",
    "                         learning_rate=0.1, \n",
    "                         max_depth=-1, \n",
    "                         num_leaves=31)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "e8eb9fb4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:39:34.135479Z",
     "iopub.status.busy": "2024-05-15T17:39:34.135023Z",
     "iopub.status.idle": "2024-05-15T17:40:33.320641Z",
     "shell.execute_reply": "2024-05-15T17:40:33.319662Z"
    },
    "papermill": {
     "duration": 59.219411,
     "end_time": "2024-05-15T17:40:33.323351",
     "exception": false,
     "start_time": "2024-05-15T17:39:34.103940",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] min_sum_hessian_in_leaf is set=1, min_child_weight=0.001 will be ignored. Current value: min_sum_hessian_in_leaf=1\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.01, subsample=1.0 will be ignored. Current value: bagging_fraction=0.01\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] min_sum_hessian_in_leaf is set=1, min_child_weight=0.001 will be ignored. Current value: min_sum_hessian_in_leaf=1\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.01, subsample=1.0 will be ignored. Current value: bagging_fraction=0.01\n",
      "[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.015600 seconds.\n",
      "You can set `force_row_wise=true` to remove the overhead.\n",
      "And if memory is not enough, you can set `force_col_wise=true`.\n",
      "[LightGBM] [Info] Total Bins 4323\n",
      "[LightGBM] [Info] Number of data points in the train set: 7645, number of used features: 183\n",
      "[LightGBM] [Info] Start training from score -1.386294\n",
      "[LightGBM] [Info] Start training from score -1.386294\n",
      "[LightGBM] [Info] Start training from score -1.386294\n",
      "[LightGBM] [Info] Start training from score -1.386294\n"
     ]
    }
   ],
   "source": [
    "model_lgbm = model_lgbm.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "965d8109",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:33.386345Z",
     "iopub.status.busy": "2024-05-15T17:40:33.385694Z",
     "iopub.status.idle": "2024-05-15T17:40:33.905174Z",
     "shell.execute_reply": "2024-05-15T17:40:33.903923Z"
    },
    "papermill": {
     "duration": 0.552071,
     "end_time": "2024-05-15T17:40:33.907730",
     "exception": false,
     "start_time": "2024-05-15T17:40:33.355659",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_sum_hessian_in_leaf is set=1, min_child_weight=0.001 will be ignored. Current value: min_sum_hessian_in_leaf=1\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.01, subsample=1.0 will be ignored. Current value: bagging_fraction=0.01\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     class 1       0.95      0.88      0.91       164\n",
      "     class 2       0.90      0.93      0.91       311\n",
      "     class 3       0.88      0.89      0.88       254\n",
      "     class 4       0.97      0.97      0.97      1183\n",
      "\n",
      "    accuracy                           0.95      1912\n",
      "   macro avg       0.92      0.92      0.92      1912\n",
      "weighted avg       0.95      0.95      0.95      1912\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, model_lgbm.predict(X_test),target_names=['class 1', 'class 2', 'class 3','class 4']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "fa8ddc78",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:33.969862Z",
     "iopub.status.busy": "2024-05-15T17:40:33.968653Z",
     "iopub.status.idle": "2024-05-15T17:40:34.473598Z",
     "shell.execute_reply": "2024-05-15T17:40:34.472364Z"
    },
    "papermill": {
     "duration": 0.539377,
     "end_time": "2024-05-15T17:40:34.476023",
     "exception": false,
     "start_time": "2024-05-15T17:40:33.936646",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_sum_hessian_in_leaf is set=1, min_child_weight=0.001 will be ignored. Current value: min_sum_hessian_in_leaf=1\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.01, subsample=1.0 will be ignored. Current value: bagging_fraction=0.01\n",
      "F1 score macro:  0.92060499879249\n"
     ]
    }
   ],
   "source": [
    "print(\"F1 score macro: \", f1_score(y_test, model_lgbm.predict(X_test), average = 'macro'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95cdba36",
   "metadata": {
    "papermill": {
     "duration": 0.029532,
     "end_time": "2024-05-15T17:40:34.534941",
     "exception": false,
     "start_time": "2024-05-15T17:40:34.505409",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### K-Nearest Neighbors Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "2f05c865",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:34.593203Z",
     "iopub.status.busy": "2024-05-15T17:40:34.592848Z",
     "iopub.status.idle": "2024-05-15T17:40:34.803563Z",
     "shell.execute_reply": "2024-05-15T17:40:34.802202Z"
    },
    "papermill": {
     "duration": 0.242951,
     "end_time": "2024-05-15T17:40:34.806388",
     "exception": false,
     "start_time": "2024-05-15T17:40:34.563437",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "a7ca597f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:34.867616Z",
     "iopub.status.busy": "2024-05-15T17:40:34.867222Z",
     "iopub.status.idle": "2024-05-15T17:40:34.872990Z",
     "shell.execute_reply": "2024-05-15T17:40:34.871477Z"
    },
    "papermill": {
     "duration": 0.038673,
     "end_time": "2024-05-15T17:40:34.875557",
     "exception": false,
     "start_time": "2024-05-15T17:40:34.836884",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_knn = Pipeline([('scaler', StandardScaler()),\n",
    "                      ('model', KNeighborsClassifier(n_neighbors=4))])\n",
    "#model_knn.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "aa6211ce",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:34.937904Z",
     "iopub.status.busy": "2024-05-15T17:40:34.937543Z",
     "iopub.status.idle": "2024-05-15T17:40:34.943221Z",
     "shell.execute_reply": "2024-05-15T17:40:34.942008Z"
    },
    "papermill": {
     "duration": 0.038383,
     "end_time": "2024-05-15T17:40:34.945851",
     "exception": false,
     "start_time": "2024-05-15T17:40:34.907468",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#print(classification_report(y_test, model_knn.predict(X_test),target_names=['class 1', 'class 2', 'class 3','class 4']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "887685ac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:35.011578Z",
     "iopub.status.busy": "2024-05-15T17:40:35.010925Z",
     "iopub.status.idle": "2024-05-15T17:40:35.015796Z",
     "shell.execute_reply": "2024-05-15T17:40:35.014667Z"
    },
    "papermill": {
     "duration": 0.041137,
     "end_time": "2024-05-15T17:40:35.017961",
     "exception": false,
     "start_time": "2024-05-15T17:40:34.976824",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#print(\"F1 score macro: \", f1_score(y_test, model_knn.predict(X_test), average = 'macro'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b34f1ca7",
   "metadata": {
    "papermill": {
     "duration": 0.030523,
     "end_time": "2024-05-15T17:40:35.080376",
     "exception": false,
     "start_time": "2024-05-15T17:40:35.049853",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Gradient Boosting Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "2c422342",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:35.142611Z",
     "iopub.status.busy": "2024-05-15T17:40:35.142182Z",
     "iopub.status.idle": "2024-05-15T17:40:35.323589Z",
     "shell.execute_reply": "2024-05-15T17:40:35.322346Z"
    },
    "papermill": {
     "duration": 0.216628,
     "end_time": "2024-05-15T17:40:35.325983",
     "exception": false,
     "start_time": "2024-05-15T17:40:35.109355",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;std_scaler&#x27;,\n",
       "                                                  StandardScaler(),\n",
       "                                                  [&#x27;v2a1&#x27;, &#x27;edjefe&#x27;, &#x27;edjefa&#x27;,\n",
       "                                                   &#x27;meaneduc&#x27;,\n",
       "                                                   &#x27;rent_per_person&#x27;]),\n",
       "                                                 (&#x27;minmax_scaler&#x27;,\n",
       "                                                  MinMaxScaler(),\n",
       "                                                  [&#x27;tipovivi4&#x27;, &#x27;escolari-std_&#x27;,\n",
       "                                                   &#x27;estadocivil2-std_&#x27;,\n",
       "                                                   &#x27;energcocinar1&#x27;,\n",
       "                                                   &#x27;estadocivil1-sum&#x27;,\n",
       "                                                   &#x27;parentesco10-mean&#x27;,\n",
       "                                                   &#x27;energcocinar3&#x27;, &#x27;dis-std_&#x27;,\n",
       "                                                   &#x27;estadocivil3-max&#x27;,\n",
       "                                                   &#x27;estadocivil6-st...\n",
       "                                                   &#x27;parentesco8-max&#x27;,\n",
       "                                                   &#x27;parentesco1-max&#x27;,\n",
       "                                                   &#x27;parentesco11-sum&#x27;,\n",
       "                                                   &#x27;SQBescolari&#x27;,\n",
       "                                                   &#x27;parentesco7-std_&#x27;,\n",
       "                                                   &#x27;television&#x27;,\n",
       "                                                   &#x27;SQBovercrowding&#x27;,\n",
       "                                                   &#x27;escolari-max&#x27;,\n",
       "                                                   &#x27;estadocivil4-sum&#x27;,\n",
       "                                                   &#x27;energcocinar4&#x27;, &#x27;elimbasu1&#x27;,\n",
       "                                                   &#x27;parentesco1-min&#x27;,\n",
       "                                                   &#x27;estadocivil7-mean&#x27;,\n",
       "                                                   &#x27;pisocemento&#x27;, &#x27;cielorazo&#x27;,\n",
       "                                                   &#x27;lugar4&#x27;, ...])])),\n",
       "                (&#x27;classifier&#x27;,\n",
       "                 GradientBoostingClassifier(learning_rate=0.3, max_depth=7,\n",
       "                                            n_estimators=200))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;std_scaler&#x27;,\n",
       "                                                  StandardScaler(),\n",
       "                                                  [&#x27;v2a1&#x27;, &#x27;edjefe&#x27;, &#x27;edjefa&#x27;,\n",
       "                                                   &#x27;meaneduc&#x27;,\n",
       "                                                   &#x27;rent_per_person&#x27;]),\n",
       "                                                 (&#x27;minmax_scaler&#x27;,\n",
       "                                                  MinMaxScaler(),\n",
       "                                                  [&#x27;tipovivi4&#x27;, &#x27;escolari-std_&#x27;,\n",
       "                                                   &#x27;estadocivil2-std_&#x27;,\n",
       "                                                   &#x27;energcocinar1&#x27;,\n",
       "                                                   &#x27;estadocivil1-sum&#x27;,\n",
       "                                                   &#x27;parentesco10-mean&#x27;,\n",
       "                                                   &#x27;energcocinar3&#x27;, &#x27;dis-std_&#x27;,\n",
       "                                                   &#x27;estadocivil3-max&#x27;,\n",
       "                                                   &#x27;estadocivil6-st...\n",
       "                                                   &#x27;parentesco8-max&#x27;,\n",
       "                                                   &#x27;parentesco1-max&#x27;,\n",
       "                                                   &#x27;parentesco11-sum&#x27;,\n",
       "                                                   &#x27;SQBescolari&#x27;,\n",
       "                                                   &#x27;parentesco7-std_&#x27;,\n",
       "                                                   &#x27;television&#x27;,\n",
       "                                                   &#x27;SQBovercrowding&#x27;,\n",
       "                                                   &#x27;escolari-max&#x27;,\n",
       "                                                   &#x27;estadocivil4-sum&#x27;,\n",
       "                                                   &#x27;energcocinar4&#x27;, &#x27;elimbasu1&#x27;,\n",
       "                                                   &#x27;parentesco1-min&#x27;,\n",
       "                                                   &#x27;estadocivil7-mean&#x27;,\n",
       "                                                   &#x27;pisocemento&#x27;, &#x27;cielorazo&#x27;,\n",
       "                                                   &#x27;lugar4&#x27;, ...])])),\n",
       "                (&#x27;classifier&#x27;,\n",
       "                 GradientBoostingClassifier(learning_rate=0.3, max_depth=7,\n",
       "                                            n_estimators=200))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">preprocessor: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;std_scaler&#x27;, StandardScaler(),\n",
       "                                 [&#x27;v2a1&#x27;, &#x27;edjefe&#x27;, &#x27;edjefa&#x27;, &#x27;meaneduc&#x27;,\n",
       "                                  &#x27;rent_per_person&#x27;]),\n",
       "                                (&#x27;minmax_scaler&#x27;, MinMaxScaler(),\n",
       "                                 [&#x27;tipovivi4&#x27;, &#x27;escolari-std_&#x27;,\n",
       "                                  &#x27;estadocivil2-std_&#x27;, &#x27;energcocinar1&#x27;,\n",
       "                                  &#x27;estadocivil1-sum&#x27;, &#x27;parentesco10-mean&#x27;,\n",
       "                                  &#x27;energcocinar3&#x27;, &#x27;dis-std_&#x27;,\n",
       "                                  &#x27;estadocivil3-max&#x27;, &#x27;estadocivil6-std_&#x27;,\n",
       "                                  &#x27;parentesco6-sum&#x27;, &#x27;parentesco2-std_&#x27;,\n",
       "                                  &#x27;paredpreb&#x27;, &#x27;SQBmeaned&#x27;, &#x27;parentesco8-max&#x27;,\n",
       "                                  &#x27;parentesco1-max&#x27;, &#x27;parentesco11-sum&#x27;,\n",
       "                                  &#x27;SQBescolari&#x27;, &#x27;parentesco7-std_&#x27;,\n",
       "                                  &#x27;television&#x27;, &#x27;SQBovercrowding&#x27;,\n",
       "                                  &#x27;escolari-max&#x27;, &#x27;estadocivil4-sum&#x27;,\n",
       "                                  &#x27;energcocinar4&#x27;, &#x27;elimbasu1&#x27;,\n",
       "                                  &#x27;parentesco1-min&#x27;, &#x27;estadocivil7-mean&#x27;,\n",
       "                                  &#x27;pisocemento&#x27;, &#x27;cielorazo&#x27;, &#x27;lugar4&#x27;, ...])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">std_scaler</label><div class=\"sk-toggleable__content\"><pre>[&#x27;v2a1&#x27;, &#x27;edjefe&#x27;, &#x27;edjefa&#x27;, &#x27;meaneduc&#x27;, &#x27;rent_per_person&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">minmax_scaler</label><div class=\"sk-toggleable__content\"><pre>[&#x27;tipovivi4&#x27;, &#x27;escolari-std_&#x27;, &#x27;estadocivil2-std_&#x27;, &#x27;energcocinar1&#x27;, &#x27;estadocivil1-sum&#x27;, &#x27;parentesco10-mean&#x27;, &#x27;energcocinar3&#x27;, &#x27;dis-std_&#x27;, &#x27;estadocivil3-max&#x27;, &#x27;estadocivil6-std_&#x27;, &#x27;parentesco6-sum&#x27;, &#x27;parentesco2-std_&#x27;, &#x27;paredpreb&#x27;, &#x27;SQBmeaned&#x27;, &#x27;parentesco8-max&#x27;, &#x27;parentesco1-max&#x27;, &#x27;parentesco11-sum&#x27;, &#x27;SQBescolari&#x27;, &#x27;parentesco7-std_&#x27;, &#x27;television&#x27;, &#x27;SQBovercrowding&#x27;, &#x27;escolari-max&#x27;, &#x27;estadocivil4-sum&#x27;, &#x27;energcocinar4&#x27;, &#x27;elimbasu1&#x27;, &#x27;parentesco1-min&#x27;, &#x27;estadocivil7-mean&#x27;, &#x27;pisocemento&#x27;, &#x27;cielorazo&#x27;, &#x27;lugar4&#x27;, &#x27;adult_dependency_rate&#x27;, &#x27;estadocivil4-max&#x27;, &#x27;parentesco10-max&#x27;, &#x27;parentesco8-mean&#x27;, &#x27;paredother&#x27;, &#x27;male-mean&#x27;, &#x27;parentesco4-mean&#x27;, &#x27;paredblolad&#x27;, &#x27;parentesco1-mean&#x27;, &#x27;parentesco3-max&#x27;, &#x27;dependency_num&#x27;, &#x27;techocane&#x27;, &#x27;parentesco3-mean&#x27;, &#x27;parentesco12-std_&#x27;, &#x27;parentesco4-min&#x27;, &#x27;elimbasu4&#x27;, &#x27;pisomadera&#x27;, &#x27;v18q1&#x27;, &#x27;instlevel-max&#x27;, &#x27;instlevel-std_&#x27;, &#x27;r4h1&#x27;, &#x27;escolari-sum&#x27;, &#x27;parentesco5-mean&#x27;, &#x27;r4m1&#x27;, &#x27;coopele&#x27;, &#x27;instlevel-min&#x27;, &#x27;tipovivi2&#x27;, &#x27;age-min&#x27;, &#x27;parentesco8-std_&#x27;, &#x27;techoentrepiso&#x27;, &#x27;pisoother&#x27;, &#x27;parentesco2-max&#x27;, &#x27;estadocivil7-std_&#x27;, &#x27;elimbasu5&#x27;, &#x27;age-sum&#x27;, &#x27;sanitario6&#x27;, &#x27;age-mean&#x27;, &#x27;age-max&#x27;, &#x27;SQBhogar_total&#x27;, &#x27;sanitario2&#x27;, &#x27;public&#x27;, &#x27;SQBedjefe&#x27;, &#x27;lugar5&#x27;, &#x27;parentesco7-max&#x27;, &#x27;instlevel-sum&#x27;, &#x27;parentesco3-std_&#x27;, &#x27;parentesco10-min&#x27;, &#x27;hacdor&#x27;, &#x27;estadocivil5-mean&#x27;, &#x27;male-sum&#x27;, &#x27;estadocivil6-sum&#x27;, &#x27;estadocivil1-min&#x27;, &#x27;dis-mean&#x27;, &#x27;dis-max&#x27;, &#x27;estadocivil6-min&#x27;, &#x27;parentesco6-mean&#x27;, &#x27;parentesco6-std_&#x27;, &#x27;parentesco12-sum&#x27;, &#x27;parentesco8-min&#x27;, &#x27;parentesco2-mean&#x27;, &#x27;rooms&#x27;, &#x27;qmobilephone&#x27;, &#x27;agesq&#x27;, &#x27;hacapo&#x27;, &#x27;male-max&#x27;, &#x27;parentesco9-std_&#x27;, &#x27;r4m2&#x27;, &#x27;SQBdependency&#x27;, &#x27;estadocivil1-mean&#x27;, &#x27;estadocivil4-mean&#x27;, &#x27;pisomoscer&#x27;, &#x27;SQBhogar_nin&#x27;, &#x27;estadocivil7-min&#x27;, &#x27;parentesco6-min&#x27;, &#x27;escolari-min&#x27;, &#x27;estadocivil5-std_&#x27;, &#x27;parentesco12-min&#x27;, &#x27;estadocivil3-std_&#x27;, &#x27;parentesco9-max&#x27;, &#x27;v14a&#x27;, &#x27;estadocivil2-sum&#x27;, &#x27;parentesco4-sum&#x27;, &#x27;estadocivil1-max&#x27;, &#x27;tipovivi3&#x27;, &#x27;eviv&#x27;, &#x27;sanitario1&#x27;, &#x27;parentesco11-max&#x27;, &#x27;parentesco9-sum&#x27;, &#x27;parentesco11-min&#x27;, &#x27;r4m3&#x27;, &#x27;lugar3&#x27;, &#x27;estadocivil3-mean&#x27;, &#x27;estadocivil7-sum&#x27;, &#x27;parentesco5-max&#x27;, &#x27;head_is_adult&#x27;, &#x27;dis-min&#x27;, &#x27;parentesco9-min&#x27;, &#x27;parentesco11-std_&#x27;, &#x27;age-std_&#x27;, &#x27;estadocivil3-sum&#x27;, &#x27;adult_rate&#x27;, &#x27;parentesco4-max&#x27;, &#x27;tipovivi1&#x27;, &#x27;elimbasu3&#x27;, &#x27;lugar6&#x27;, &#x27;techootro&#x27;, &#x27;children_rate&#x27;, &#x27;sanitario5&#x27;, &#x27;parentesco7-mean&#x27;, &#x27;parentesco12-mean&#x27;, &#x27;escolari-mean&#x27;, &#x27;estadocivil1-std_&#x27;, &#x27;estadocivil4-min&#x27;, &#x27;paredmad&#x27;, &#x27;parentesco7-min&#x27;, &#x27;parentesco11-mean&#x27;, &#x27;planpri&#x27;, &#x27;r4h2&#x27;, &#x27;hogar_total&#x27;, &#x27;estadocivil2-max&#x27;, &#x27;estadocivil5-sum&#x27;, &#x27;estadocivil4-std_&#x27;, &#x27;noelec&#x27;, &#x27;parentesco5-min&#x27;, &#x27;parentesco5-std_&#x27;, &#x27;adult_num&#x27;, &#x27;bedroom_per_person&#x27;, &#x27;parentesco3-sum&#x27;, &#x27;parentesco8-sum&#x27;, &#x27;techozinc&#x27;, &#x27;parentesco2-min&#x27;, &#x27;etecho&#x27;, &#x27;paredzocalo&#x27;, &#x27;parentesco6-max&#x27;, &#x27;estadocivil6-mean&#x27;, &#x27;male-min&#x27;, &#x27;dis-sum&#x27;, &#x27;elimbasu6&#x27;, &#x27;parentesco10-sum&#x27;, &#x27;parentesco12-max&#x27;, &#x27;bedrooms&#x27;, &#x27;estadocivil5-min&#x27;, &#x27;elimbasu2&#x27;, &#x27;energcocinar2&#x27;, &#x27;hogar_adul&#x27;, &#x27;estadocivil2-mean&#x27;, &#x27;parentesco4-std_&#x27;, &#x27;parentesco9-mean&#x27;, &#x27;parentesco1-std_&#x27;, &#x27;paredzinc&#x27;, &#x27;computer&#x27;, &#x27;rez_esc&#x27;, &#x27;hogar_nin&#x27;, &#x27;elder_rate&#x27;, &#x27;lugar2&#x27;, &#x27;rent_per_adult&#x27;, &#x27;lugar1&#x27;, &#x27;estadocivil5-max&#x27;, &#x27;refrig&#x27;, &#x27;pisonotiene&#x27;, &#x27;estadocivil2-min&#x27;, &#x27;epared&#x27;, &#x27;estadocivil3-min&#x27;, &#x27;parentesco3-min&#x27;, &#x27;abastaguadentro&#x27;, &#x27;parentesco10-std_&#x27;, &#x27;dependency&#x27;, &#x27;estadocivil7-max&#x27;, &#x27;parentesco5-sum&#x27;, &#x27;bedroom_per_adult&#x27;, &#x27;pareddes&#x27;, &#x27;abastaguano&#x27;, &#x27;area1&#x27;, &#x27;dependency_rate&#x27;, &#x27;pisonatur&#x27;, &#x27;paredfibras&#x27;, &#x27;abastaguafuera&#x27;, &#x27;tipovivi5&#x27;, &#x27;r4h3&#x27;, &#x27;hogar_mayor&#x27;, &#x27;sanitario3&#x27;, &#x27;overcrowding&#x27;, &#x27;parentesco7-sum&#x27;, &#x27;male-std_&#x27;, &#x27;estadocivil6-max&#x27;, &#x27;instlevel-mean&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MinMaxScaler</label><div class=\"sk-toggleable__content\"><pre>MinMaxScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GradientBoostingClassifier</label><div class=\"sk-toggleable__content\"><pre>GradientBoostingClassifier(learning_rate=0.3, max_depth=7, n_estimators=200)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('preprocessor',\n",
       "                 ColumnTransformer(transformers=[('std_scaler',\n",
       "                                                  StandardScaler(),\n",
       "                                                  ['v2a1', 'edjefe', 'edjefa',\n",
       "                                                   'meaneduc',\n",
       "                                                   'rent_per_person']),\n",
       "                                                 ('minmax_scaler',\n",
       "                                                  MinMaxScaler(),\n",
       "                                                  ['tipovivi4', 'escolari-std_',\n",
       "                                                   'estadocivil2-std_',\n",
       "                                                   'energcocinar1',\n",
       "                                                   'estadocivil1-sum',\n",
       "                                                   'parentesco10-mean',\n",
       "                                                   'energcocinar3', 'dis-std_',\n",
       "                                                   'estadocivil3-max',\n",
       "                                                   'estadocivil6-st...\n",
       "                                                   'parentesco8-max',\n",
       "                                                   'parentesco1-max',\n",
       "                                                   'parentesco11-sum',\n",
       "                                                   'SQBescolari',\n",
       "                                                   'parentesco7-std_',\n",
       "                                                   'television',\n",
       "                                                   'SQBovercrowding',\n",
       "                                                   'escolari-max',\n",
       "                                                   'estadocivil4-sum',\n",
       "                                                   'energcocinar4', 'elimbasu1',\n",
       "                                                   'parentesco1-min',\n",
       "                                                   'estadocivil7-mean',\n",
       "                                                   'pisocemento', 'cielorazo',\n",
       "                                                   'lugar4', ...])])),\n",
       "                ('classifier',\n",
       "                 GradientBoostingClassifier(learning_rate=0.3, max_depth=7,\n",
       "                                            n_estimators=200))])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.utils import class_weight\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('std_scaler', StandardScaler(), col_std),  # chuẩn hóa các cột trong nhóm col_std\n",
    "        ('minmax_scaler', MinMaxScaler(), col_minmax)  # chuẩn hóa các cột trong nhóm col_minmax\n",
    "    ])\n",
    "\n",
    "gradient = GradientBoostingClassifier()\n",
    "\n",
    "model_gbc = Pipeline([\n",
    "    ('preprocessor', preprocessor),  # chuẩn hóa dữ liệu\n",
    "    ('classifier', gradient)  # mô hình phân loại\n",
    "])\n",
    "\n",
    "best_param =  {'classifier__learning_rate': 0.3, \n",
    "               'classifier__max_depth': 7, \n",
    "               'classifier__n_estimators': 200}\n",
    "\n",
    "\n",
    "model_gbc.set_params(**best_param) # set siêu tham số cho mô hình"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "99ca1643",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:35.388896Z",
     "iopub.status.busy": "2024-05-15T17:40:35.388478Z",
     "iopub.status.idle": "2024-05-15T17:40:35.393313Z",
     "shell.execute_reply": "2024-05-15T17:40:35.392401Z"
    },
    "papermill": {
     "duration": 0.039407,
     "end_time": "2024-05-15T17:40:35.395582",
     "exception": false,
     "start_time": "2024-05-15T17:40:35.356175",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#y_weights = class_weight.compute_sample_weight('balanced', y, indices=None)\n",
    "#X_train,X_test,y_train,y_test, weights_train,weights_test = train_test_split(X,y,y_weights,test_size=0.2, random_state=1)\n",
    "\n",
    "#model_gbc.fit(X_train, y_train, classifier__sample_weight=weights_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "da07a209",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:35.456448Z",
     "iopub.status.busy": "2024-05-15T17:40:35.456014Z",
     "iopub.status.idle": "2024-05-15T17:40:35.460379Z",
     "shell.execute_reply": "2024-05-15T17:40:35.459381Z"
    },
    "papermill": {
     "duration": 0.036939,
     "end_time": "2024-05-15T17:40:35.462662",
     "exception": false,
     "start_time": "2024-05-15T17:40:35.425723",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#print(classification_report(y_test, model_gbc.predict(X_test),target_names=['class 1', 'class 2', 'class 3','class 4']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "d1e7aae5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:35.524268Z",
     "iopub.status.busy": "2024-05-15T17:40:35.523832Z",
     "iopub.status.idle": "2024-05-15T17:40:35.529547Z",
     "shell.execute_reply": "2024-05-15T17:40:35.527662Z"
    },
    "papermill": {
     "duration": 0.040996,
     "end_time": "2024-05-15T17:40:35.532719",
     "exception": false,
     "start_time": "2024-05-15T17:40:35.491723",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#print(\"F1 score macro: \", f1_score(y_test, model_gbc.predict(X_test), average = 'macro'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf4d8e50",
   "metadata": {
    "papermill": {
     "duration": 0.031151,
     "end_time": "2024-05-15T17:40:35.593696",
     "exception": false,
     "start_time": "2024-05-15T17:40:35.562545",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### XGBoost Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "7aff1c9c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:35.656234Z",
     "iopub.status.busy": "2024-05-15T17:40:35.655793Z",
     "iopub.status.idle": "2024-05-15T17:40:35.873422Z",
     "shell.execute_reply": "2024-05-15T17:40:35.872236Z"
    },
    "papermill": {
     "duration": 0.253184,
     "end_time": "2024-05-15T17:40:35.876117",
     "exception": false,
     "start_time": "2024-05-15T17:40:35.622933",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "\n",
    "model_xgb = XGBClassifier(\n",
    "    booster='gbtree',\n",
    "    learning_rate=0.1,\n",
    "    n_estimators=1200,\n",
    "    max_depth=6,\n",
    "    min_child_weight=1,\n",
    "    gamma=0,\n",
    "    subsample=0.8,\n",
    "    colsample_bytree=0.8,\n",
    "    objective='multi:softmax',\n",
    "    num_class=4,\n",
    "    reg_alpha=0,\n",
    "    reg_lambda=1,\n",
    "    random_state=42,\n",
    "    verbosity=0\n",
    ") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "34b042a9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:35.939224Z",
     "iopub.status.busy": "2024-05-15T17:40:35.938825Z",
     "iopub.status.idle": "2024-05-15T17:40:35.944678Z",
     "shell.execute_reply": "2024-05-15T17:40:35.943374Z"
    },
    "papermill": {
     "duration": 0.041178,
     "end_time": "2024-05-15T17:40:35.947650",
     "exception": false,
     "start_time": "2024-05-15T17:40:35.906472",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#model_xgb.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "997bbecc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:36.017047Z",
     "iopub.status.busy": "2024-05-15T17:40:36.016629Z",
     "iopub.status.idle": "2024-05-15T17:40:36.021728Z",
     "shell.execute_reply": "2024-05-15T17:40:36.020836Z"
    },
    "papermill": {
     "duration": 0.04437,
     "end_time": "2024-05-15T17:40:36.024115",
     "exception": false,
     "start_time": "2024-05-15T17:40:35.979745",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#print(classification_report(y_test, model_xgb.predict(X_test), target_names=['class 1', 'class 2', 'class 3', 'class 4']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "252caf2d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:36.090385Z",
     "iopub.status.busy": "2024-05-15T17:40:36.089918Z",
     "iopub.status.idle": "2024-05-15T17:40:36.094713Z",
     "shell.execute_reply": "2024-05-15T17:40:36.093561Z"
    },
    "papermill": {
     "duration": 0.039957,
     "end_time": "2024-05-15T17:40:36.097173",
     "exception": false,
     "start_time": "2024-05-15T17:40:36.057216",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#print(\"F1 score macro: \", f1_score(y_test, model_xgb.predict(X_test), average = 'macro'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc3f6b7e",
   "metadata": {
    "papermill": {
     "duration": 0.030544,
     "end_time": "2024-05-15T17:40:36.159961",
     "exception": false,
     "start_time": "2024-05-15T17:40:36.129417",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Random Forest Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "da83aa90",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:36.227939Z",
     "iopub.status.busy": "2024-05-15T17:40:36.227517Z",
     "iopub.status.idle": "2024-05-15T17:40:36.235724Z",
     "shell.execute_reply": "2024-05-15T17:40:36.233924Z"
    },
    "papermill": {
     "duration": 0.046533,
     "end_time": "2024-05-15T17:40:36.238243",
     "exception": false,
     "start_time": "2024-05-15T17:40:36.191710",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "model_rfc = Pipeline([\n",
    "    ('scaler', MinMaxScaler()),\n",
    "    ('model', RandomForestClassifier(n_estimators=100,\n",
    "                                     criterion='log_loss',\n",
    "                                     max_features=None,\n",
    "                                     class_weight='balanced_subsample',)),])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "8ae02103",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:36.300206Z",
     "iopub.status.busy": "2024-05-15T17:40:36.299755Z",
     "iopub.status.idle": "2024-05-15T17:40:36.304516Z",
     "shell.execute_reply": "2024-05-15T17:40:36.303645Z"
    },
    "papermill": {
     "duration": 0.038379,
     "end_time": "2024-05-15T17:40:36.306744",
     "exception": false,
     "start_time": "2024-05-15T17:40:36.268365",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#model_rfc.fit(X_train, y_train, **{'model__sample_weight': weights_train})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "9d58ca37",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:36.370591Z",
     "iopub.status.busy": "2024-05-15T17:40:36.370029Z",
     "iopub.status.idle": "2024-05-15T17:40:36.374542Z",
     "shell.execute_reply": "2024-05-15T17:40:36.373645Z"
    },
    "papermill": {
     "duration": 0.038634,
     "end_time": "2024-05-15T17:40:36.376627",
     "exception": false,
     "start_time": "2024-05-15T17:40:36.337993",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#print(classification_report(y_test, model_rfc.predict(X_test), target_names=['class 1', 'class 2', 'class 3', 'class 4']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "0a75bc02",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:36.440467Z",
     "iopub.status.busy": "2024-05-15T17:40:36.440058Z",
     "iopub.status.idle": "2024-05-15T17:40:36.444306Z",
     "shell.execute_reply": "2024-05-15T17:40:36.443395Z"
    },
    "papermill": {
     "duration": 0.037947,
     "end_time": "2024-05-15T17:40:36.446509",
     "exception": false,
     "start_time": "2024-05-15T17:40:36.408562",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#print(\"F1 score macro: \", f1_score(y_test, model_rfc.predict(X_test), average = 'macro'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5d0b931",
   "metadata": {
    "papermill": {
     "duration": 0.02908,
     "end_time": "2024-05-15T17:40:36.506114",
     "exception": false,
     "start_time": "2024-05-15T17:40:36.477034",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Submissions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "183a72fc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:36.566992Z",
     "iopub.status.busy": "2024-05-15T17:40:36.566583Z",
     "iopub.status.idle": "2024-05-15T17:40:42.917136Z",
     "shell.execute_reply": "2024-05-15T17:40:42.915915Z"
    },
    "papermill": {
     "duration": 6.384511,
     "end_time": "2024-05-15T17:40:42.919778",
     "exception": false,
     "start_time": "2024-05-15T17:40:36.535267",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_sum_hessian_in_leaf is set=1, min_child_weight=0.001 will be ignored. Current value: min_sum_hessian_in_leaf=1\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.01, subsample=1.0 will be ignored. Current value: bagging_fraction=0.01\n"
     ]
    }
   ],
   "source": [
    "lgbm_submission = pd.DataFrame({'Id': data_test_id, 'Target': model_lgbm.predict(data_test)+1})\n",
    "lgbm_submission.to_csv('submission.csv', index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "1deba4f1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:42.981234Z",
     "iopub.status.busy": "2024-05-15T17:40:42.980871Z",
     "iopub.status.idle": "2024-05-15T17:40:42.985241Z",
     "shell.execute_reply": "2024-05-15T17:40:42.984033Z"
    },
    "papermill": {
     "duration": 0.038703,
     "end_time": "2024-05-15T17:40:42.987802",
     "exception": false,
     "start_time": "2024-05-15T17:40:42.949099",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#knn_submission = pd.DataFrame({'Id': data_test_id, 'Target': model_knn.predict(data_test)+1})\n",
    "#knn_submission.to_csv('submission.csv', index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "a436b982",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:43.048852Z",
     "iopub.status.busy": "2024-05-15T17:40:43.048488Z",
     "iopub.status.idle": "2024-05-15T17:40:43.054440Z",
     "shell.execute_reply": "2024-05-15T17:40:43.053435Z"
    },
    "papermill": {
     "duration": 0.039179,
     "end_time": "2024-05-15T17:40:43.056739",
     "exception": false,
     "start_time": "2024-05-15T17:40:43.017560",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#gbc_submission = pd.DataFrame({'Id': data_test_id, 'Target': model_gbc.predict(data_test)+1})\n",
    "#gbc_submission.to_csv('submission.csv', index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "e7cc5a86",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:43.117480Z",
     "iopub.status.busy": "2024-05-15T17:40:43.117113Z",
     "iopub.status.idle": "2024-05-15T17:40:43.121476Z",
     "shell.execute_reply": "2024-05-15T17:40:43.120358Z"
    },
    "papermill": {
     "duration": 0.037592,
     "end_time": "2024-05-15T17:40:43.123822",
     "exception": false,
     "start_time": "2024-05-15T17:40:43.086230",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#xgb_submission = pd.DataFrame({'Id': data_test_id, 'Target': model_xgb.predict(data_test)+1})\n",
    "#xgb_submission.to_csv('submission.csv', index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "1b9f6009",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-15T17:40:43.186080Z",
     "iopub.status.busy": "2024-05-15T17:40:43.185706Z",
     "iopub.status.idle": "2024-05-15T17:40:43.191000Z",
     "shell.execute_reply": "2024-05-15T17:40:43.189452Z"
    },
    "papermill": {
     "duration": 0.040827,
     "end_time": "2024-05-15T17:40:43.193467",
     "exception": false,
     "start_time": "2024-05-15T17:40:43.152640",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#rfc_submission = pd.DataFrame({'Id': data_test_id, 'Target': model_rfc.predict(data_test)+1})\n",
    "#rfc_submission.to_csv('submission.csv', index = False)"
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "none",
   "dataSources": [
    {
     "databundleVersionId": 59638,
     "sourceId": 9840,
     "sourceType": "competition"
    }
   ],
   "dockerImageVersionId": 30698,
   "isGpuEnabled": false,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 105.228157,
   "end_time": "2024-05-15T17:40:44.047248",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2024-05-15T17:38:58.819091",
   "version": "2.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
